IJHCS 2026 | XR Agents Need More Than Chat — First Sense Who Wants to Speak

Someone stands before a virtual post-office receptionist but has not spoken yet. Are they queuing, merely looking around, or already waiting for help? If the system speaks too early, it feels intrusive; if it waits for an explicit request, it loses the point of proactive service.

That moment exposes what makes XR agents harder than chatbots. They must not only generate a fluent sentence; they must see people, distinguish groups, guess interaction intent, read emotion, then decide when to speak, whom to address, and how the body should move while speaking.

This IJHCS paper is worth reading not because it merely plugs another LLM into XR. More important is that the authors break social interaction into replaceable, measurable system stages. Social acceptability is not a layer of packaging on character looks or tone — it has to enter the architecture itself.

01 What the agent must solve first is not how to answer, but when it should answer

The authors draw one interaction as a loop: the system first perceives social signals, then interprets intent, emotion, and group dynamics, makes decisions from environmental and social information, and only then generates verbal and nonverbal responses. As Figure 1 shows, the language model sits only in the latter half of the chain.

That order matters. People in public places do not announce interaction start with a click the way they open a chat window. They may pass by, watch, travel together, or in a group only one representative may speak. Without intent and group judgment, even a good answer may land at the wrong moment, toward the wrong person.

The paper therefore redefines social acceptability here: not only whether users are willing to use the system, but whether interaction is satisfactory, transparent, safe, and trustworthy, and whether it fits the present social situation. That definition turns a metric often left to post-hoc questionnaires into a design condition upfront.

02 Splitting the system into modules is how social capability becomes truly replaceable

The full architecture includes session, detection, communication, XR, user-interface, and robot modules, connected underneath through an API layer. Figure 2 shows internal modules on the left and interaction between user and XR agent on the right. It also uses OpenAPI and AsyncAPI to specify endpoints and messages so modules need not be locked to one programming language or device.

This split solves more than engineering maintenance. A post-office assistant, robot receptionist, and journalist safety trainer need different sensors, privacy constraints, body actions, and dialogue knowledge. If every capability is bound into one black box, developers can hardly replace an emotion recognizer or speech service alone while keeping other stages.

Of course, modularity does not automatically equal trustworthiness. It only draws observable boundaries around problems: whether intent detection failed or language generation was slow; where privacy data are processed; what modules exchange. Only when those boundaries can be measured and audited does social acceptability stop at design slogans.

03 Perception is not recognizing a face — it is judging how a social scene is unfolding

The paper gathers the most core functions into two kinds: detection and communication. In Figure 3, detection splits into three branches — interaction intent, groups, and user emotion; communication splits into verbal and nonverbal responses. This is not a feature checklist; it is a causal order of reading the scene first, then responding.

Intent detection uses time series of body motion, spatial position, and gaze, rather than waiting for an explicit command. Group detection tries to identify who is interacting and who is more likely the dialogue representative; multimodal emotion analysis merges facial expression, vocal tone, and spoken content. Figure 4 places these three implementation routes together.

Here is a risk easy to miss: perception modules compress uncertain social signals into system facts. Gaze may mean help-seeking or merely curiosity; facial expression is shaped by culture, setting, and individual difference. So the more practical benefit of modularity is being able to check separately under what conditions these inferences fail — not packaging them directly as intelligence.

04 The LLM decides what to say — the body still has to show how it listens

The communication module is also not reduced to text in and text out. The language side uses speech recognition, LLM, and speech synthesis, and folds user feedback, emotion, and demographic attributes into answer generation; the nonverbal side generates corresponding gestures from text and audio so the virtual body does not sit like a still sticker while chatting. Figure 5 summarizes this route.

The authors also specially handle listening actions. In real conversation, listeners nod, turn, and adjust gaze — reactions that already happen before speech finishes. For embodied agents, sitting motionless makes users doubt whether the system heard them. So social-interaction feedback speed cannot be measured only by full-answer generation time; bodily reaction itself keeps telling users that interaction is still underway.

That design also raises consistency requirements. Language may express welcome while face or gesture is slow or abrupt — possibly more unsettling than having no virtual body. The paper validates module-level metrics, but how cross-modal contradiction affects trust remains a question needing finer experiments.

05 A 90.1% task completion rate is worth noting — a 4.6-second wait should not be skipped

The authors validate the architecture at module and system levels. Intent detection success rate was 0.86, F1 0.89, AUROC between 0.80 and 0.91; group detection’s F1 gains over listed baselines were 10.2%–26.4%. In related emotion-perception work, social-acceptability score was 4.38 on a 1–5 scale.

End-to-end system validation gives more intuitive numbers: task completion rate 90.1%, system failure rate 0%, interaction-intent detection accuracy 86%, detection F1 89%. Language answers averaged 4.6 seconds; touch or keyboard interface response needed 3.1 seconds — both within the authors’ 5-second acceptability ceiling.

But “within threshold” does not mean users feel no wait. 4.6 seconds already nears the ceiling; in simple queries, urgent guidance, and multi-person turn-taking, the same duration may not feel the same. Nonverbal feedback helps here: it can fill the gap before a semantic answer appears, but the paper does not directly prove that fill eliminates waiting sensation in every situation.

06 Three scenarios show it can run — not that it can enter every public space

The system was placed in three scenarios: a receptionist collaborating with a mobile robot, a post-office assistant handling customer business, and an XR trainer helping journalists and media workers rehearse safety procedures. The robot-reception scenario had 45 participants and a questionnaire with 57 five-point items; the safety-training scenario had 25 professional participants, using a 51-item scale and qualitative interviews.

These data sit closer to real interaction than module benchmarks alone, because the study also recorded task progress, interaction duration, speech or text modality, and distance between user and robot. Still, the paper’s quantitative results pool multiple earlier module studies and scenario validations — not one large controlled experiment with a unified sample and protocol.

The authors also state the generalization boundary clearly: further validation is needed in denser, noisier, more culturally varied environments with more complex accessibility constraints; long-term acceptance and stability still lack evidence; and there is currently no direct end-to-end comparison baseline for a full XR-agent pipeline. So the paper more strongly shows that “a runnable, decomposable-to-validate architecture has taken shape,” not that “a general social XR agent has been solved.”

07 What to take away is putting social risk into interfaces and metrics

For developers, the most practical part of this paper is not a specific emotion model or TTS service, but a way of decomposing the system. Before placing an agent in a campus, hospital, transit hub, or government hall, a team can first ask: how does the system judge that someone wants to interact? When many people appear at once, whom does it face? Which data must stay at the edge? When language and body disagree, how should it degrade?

The paper’s implementation anonymizes user data; body joints and facial keypoints are not bound to real identity, and are not kept after interaction ends. Still, the authors list edge processing, data minimization, and multimodal-pipeline transparency as follow-on challenges. They do not overclaim here: modularity can clarify responsibility boundaries, but will not automatically erase privacy, bias, and safety problems.

Once an agent has a body, occupies space, and actively interrupts people, “can it speak” is only the starting point. The harder design question is: does it know when it should not speak. Paper information

Original title: Towards socially acceptable XR agents: A modular approach for user interaction

Authors: Silvia Proia, Luca Capra, Oya Celiktutan, et al. First and corresponding author Silvia Proia is at the University of Modena and Reggio Emilia; collaborating institutions also include Spindox Labs, King’s College London, IDSIA/USI-SUPSI, and Technical University of Darmstadt.

Journal: International Journal of Human–Computer Studies, 217, 103907 (2026)

Paper link: https://doi.org/10.1016/j.ijhcs.2026.103907

Key terms

XR agent: A digital entity that interacts with people through a visible form in virtual, augmented, or mixed reality. Body, position, and gesture all participate in forming interaction meaning.

Social acceptability: In this paper, not only willingness to use, but also whether interaction is transparent, safe, and trustworthy, and whether it fits the social situation.

Multimodal interaction: The system jointly processes signals such as speech, text, gaze, body motion, and facial expression — not text dialogue alone.

References

Proia, S., Capra, L., Celiktutan, O., et al. (2026). Towards socially acceptable XR agents: A modular approach for user interaction. International Journal of Human–Computer Studies, 217, 103907. https://doi.org/10.1016/j.ijhcs.2026.103907