CHI 2026 | To Make a Wall Obey, Why Must It Be Allowed to Guess What You Mean?

Someone reaches toward a wall, palm outward. Do they want the rising shelf to stop, or keep rising to hand height? Or maybe they just took something and want the shelf to retract. In this paper’s experiment, the same action did carry these different meanings — plus a fourth: having the wall grow a new shelf beside the hand.

The wall can already move. The researchers’ prototype can bend a flat surface outward into a shelf, then pull it back into the wall. But once people actually come over holding things to use it, operation is less tidy: some point while speaking, some push directly, and some hang an object in midair waiting for the wall to catch it.

This CHI 2026 study from a Carnegie Mellon team recorded that ambiguous communication. Here “understanding” was temporarily done by researchers behind the scenes — not yet an automatic recognition system. The experiment left a very practical problem: people want the wall to obey them, yet are unwilling to spell out every step.

01 First let people place the blocks — then you can see what sits beyond commands

This prototype wall has three independently movable columns that travel up and down a two-meter track. Changing internal mechanical constraints bends the flat structure into an outward protrusion that can hold everyday objects such as books and backpacks. The hardware foundation comes from the team’s prior work; this paper mainly asks how it should cooperate with people.

The authors first brought in 20 participants with architecture or design backgrounds for four speculative design workshops. People watched prototype demos, handled manually deformable models, then drew imagined use scenarios. Afterward, another 12 participants each entered the lab for an hour to complete a set of tasks placing and retrieving colored blocks.

The second study did not first teach a gesture set. Participants only received cards marked with target positions and had to figure out how to make the wall extend shelves, adjust height, hold blocks, and finally return to flat. In Figure 1, A–C are standing and seated operation; D–G show the wall retracting; bottom H is the behind-the-scenes operator’s control interface; I is the task card.

This interface states the study’s key condition: participants were told the wall could see, hear, and feel them; what actually observed and judged was researchers, who then remotely controlled the wall’s motion. That is the Wizard-of-Oz method. It lets you observe how people would use a system that seems to understand them before automatic recognition is ready.

When designing gestures against two before-and-after state diagrams, people only need to explain how to complete one change. Placing blocks produces continuous action: first find a place to put something, discover it is too high, then ask it to lower a bit. The study can therefore see how a request is supplemented and revised. Whether machines can recognize these meanings, this experiment cannot yet answer.

02 What “a bit lower” leaves out is exactly what the system must know

There was a voice-only phase in the experiment. Even under that condition, 38% of new-shelf commands did not specify which column to operate; for adjusting shelves, the share that left the target unspecified reached 70%. Both denominators are voice commands of the corresponding type — not participants, and not system error rates.

Saying you want to place a box on the right, then saying “a bit lower,” is not hard for a person to understand. The later sentence omits the target because the earlier one already set it. Behind-the-scenes operators used prior exchange, body orientation, and other information to complete the meaning. If the system kept only the current sentence, even the finest speech recognition could not recover the dropped context.

Gestures likewise cannot be read apart from surrounding actions. Figure 2 summarizes common gesture and touch actions when making shelves appear, adjust, and disappear — pointing to a location, showing an object, pushing a shelf edge, waving at a distance. Numbers in parentheses are action occurrence counts; they cannot be read as how many people preferred that operation.

Someone wanting a new shelf might hold a block in front of the wall and wait for it to come near; a seated person who cannot reach a high object might reach out in a grasping motion, expecting the wall to lower it. The palm-outward gesture mentioned at the opening might also mean stop, retract, appear, or rise. Giving each of these four meanings a fixed gesture may not match the operating habits participants have already formed.

From this the authors propose that the system continuously maintain a judgment about the user’s current goal: where they just pointed, whether the object is still there, what this next “okay” is confirming. This intent model is still a proposal — not yet implemented or validated. How much the machine must guess before it can start moving, the authors also do not give a threshold ready to adopt.

03 After trying new modes, some people returned to hands only

The experiment had four phases: free choice at the start and end, and middle phases restricted to voice only or hands only, in random order. That arrangement gave participants a chance to try modes they had not thought of at first. At the end, 8 of 12 showed a clear multimodal preference; the other 4 leaned toward voice only or body action only.

That does not mean eight people spoke and gestured at once throughout. Multimodality can be simultaneous — for example saying “here” while pointing at the wall — or a chain of actions: first point at a high object, say to lower it, take it away, then tap the shelf to retract it. When hands are already full, voice can pick up the next step without needing to free a hand first.

Change also went the other way. One participant who had mixed modes later abandoned voice, because saying how many notches to descend required counting in their head. Others at first refused to touch at all, worrying the mechanical structure was unstable. Looking only at first attempts, it is easy to treat guesses about device capability as a person’s fixed preference.

Figure 3 uses two kinds of “modality entropy” to describe this difference. The left panel compares voice versus body-action distributions across early and late phases; the right panel separates two things: whether a person mixes channels, and how rich their actions are within the same channel. Someone who only speaks may still have very rich ways of expressing. Values in the figure are standardized; negative values do not mean negative entropy occurred.

Here high entropy should not be treated as a high score. It measures richness of behavioral distribution; it does not prove greater efficiency, satisfaction, or safety; some of the computation also depends on semantic embeddings and dimensionality-reduction settings. Plus everyone went through forced middle-phase attempts, so 8 people choosing multimodal cannot imply that adding three input modes to a product will necessarily improve experience.

04 Once the wall starts caring for people, it also starts deciding for them

Back to the earlier workshops, participants’ wishes were quite concrete: when a small apartment does not need a table, put it away; when a wheelchair user needs a work surface, lower it; when a kitchen wall is busy, provide support. From the materials the authors distilled 11 values; Figure 4 places functional asks such as saving space alongside trust, autonomy, comfort, and similar asks. It is not a priority ranking produced by user voting.

Trouble often sits in seemingly considerate small decisions. In the experiment, some people hoped that as soon as an object was taken, the shelf would retract automatically. In discussion the authors then give a counterexample about watering plants: someone may only move a pot to the sink and plan to put it back later. Detecting an empty shelf is easy to describe; judging whether that shelf is finished requires knowing the person’s subsequent plan.

Private homes versus public places also turn “who to obey” into different questions. Workshop participants worried that public benches that retract automatically at night might seem to raise space utilization, yet could also be used to exclude people experiencing homelessness. That is their vigilance about future uses — not that the team deployed such benches — but it reminds designers: managers’ convenience and users’ convenience need not align.

The experiment deliberately added slight delay and wrong heights; in interviews participants did not find those errors very annoying — but some were startled by shelves suddenly extending. A correctable small error and an object suddenly approaching the body do not bring the same pressure. Asking only whether people were satisfied after the task may miss the latter moments.

I would ground the next design work in these concrete moments: which actions can try first, which must first let people notice, which must wait for clear confirmation. When an object is not yet stable, or someone is approaching, I would first see whether people can stop the motion before talking about saving commands. This paper has not yet tested these physically risky scenarios.

05 An hour of placement tasks is still far from something you can live with

The prototype also cannot pass the load-bearing gate. Very common workshop ideas were seats, benches, and support surfaces that can take the body — but this wall cannot bear human weight. Being able to hold books and backpacks still leaves a technical gap before someone can sit; fall protection imagined in sketches cannot be presented as an already realized care function.

Accessibility scenarios need the same caution. Workshop participants who raised these ideas did not have disability lived experience or related formal training; having experiment participants operate while seated cannot substitute for co-designing with disabled users. The authors explicitly recommend that follow-up make these groups co-designers, rather than only placing them in sketches of future use scenarios.

Both samples were under age 40, and workshops covered only architecture and design backgrounds. Placing blocks within an hour can expose gesture ambiguity, but cannot say how a family living there for half a year would negotiate operating rights, or whether people would still be willing to talk to a wall after novelty fades. The paper has no long-term adaptation data.

There is also a limit closer to the results themselves: how behind-the-scenes operators understand hesitation and control motion rhythm feeds back into what participants are willing to keep trying. The authors constrained that difference with a graphical interface of preset actions, but cannot fully exclude it. After the same sensors connect to an automatic model, whether interaction preferences still hold needs to be observed again.

06 The next prototype should let people see how far it has understood

Participants raised a simple ask: while the wall adjusts internal structure and has not yet moved, can lights tell people it is still listening? Another confusion was concrete: saying “left” — is that the person’s left, or the wall’s left? On this kind of device, whether input was received and whether the target was chosen correctly cannot rely on people staring at the final motion to guess.

Along these observations, I would add a few tests to the next prototype: how to correct after choosing the wrong target, whether motion can be stopped mid-move, whether a shelf should retract after an object is taken. Besides recording whether tasks finished, look back at when people noticed the system misunderstood, what they used to correct, and whether they stopped the motion in time. This paper has not yet evaluated with these metrics — that is the part I hope to keep seeing.

The authors also propose that the environment gradually learn habits users teach it, rather than seeming to know everything on day one of installation. After teaching a habit, people must also be able to withdraw it. For example, auto-retracting empty shelves was once convenient; after a workflow change it gets in the way — there should be a way to revoke it without arguing with the wall every time.

After reading this study, the demo I most want to see first is whether, after the wall picks the wrong shelf, people can easily change it back. Extending a shelf already shows what the mechanical structure can do; once it is in a room with plant pots, laptops, and people walking back and forth, allowing people to change their minds should also count as a basic capability.

Paper information

Original title: Towards Fluent Interaction with Cyber-Physical Architecture. First author Jesse T Gonzalez; the team is mainly from Carnegie Mellon University; author Neeta M Khanuja is also affiliated with ITI/Larsys at Instituto Superior Tecnico, Portugal. Published at CHI 2026. Paper link: https://doi.org/10.1145/3772318.3791787

Key terms

Cyber-Physical Architecture: Putting sensing, computation, and movable physical structure into the built environment so components such as walls can respond to people and change shape. Wizard-of-Oz: Researchers behind the scenes simulate system capabilities not yet automated, to observe how users would interact. Modality entropy: Used in this paper to describe distributional diversity between voice and body actions, and within action types — not a score of interaction effectiveness.

References

Gonzalez, J. T., Khanuja, N. M., Li, M. M., Guo, M., Olaitan, L., Lau, E., Pugh, J., Ion, A., & Hudson, S. E. (2026). Towards fluent interaction with cyber-physical architecture. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI ’26) (pp. 1–23). Association for Computing Machinery. https://doi.org/10.1145/3772318.3791787