ETRA 2026 | When an AI Avatar Looks at You, Users Instinctively Look Away

When we chat with AI, we usually care whether it is right, fast, and human-sounding. But if the AI has a face, another question quietly appears: is it “looking” at you?

That sounds like a VR or robotics problem. In practice, many AI avatars first appear on ordinary 2D screens — customer service, virtual teachers, health assistants, companion apps. Do users really care whether those on-screen eyes react?

This ETRA 2026 paper’s answer is subtle: users may not notice the difference, but eye-tracking data will. When an avatar can establish mutual gaze from the user’s look, users enter mutual gaze more often — and also look away sooner and for shorter stretches.

Paper information

Original title: Investigating Mutual Gaze in Real-Time LLM-Driven Avatar Interaction on 2D Screens

Authors and affiliations: Heiko Drewes, Abdullatif Dinc, Florian Alt, LMU Munich.

Venue: Proceedings of the ACM on Human-Computer Interaction, Vol. 10, No. 3, Article ETRA005, May 2026.

Paper link: https://doi.org/10.1145/3806019

What the paper really asks: does eye contact still have social force on a 2D screen?

In human communication, gaze is not decoration. It can signal attention, trust, and closeness — and also create pressure. Too little eye contact looks cold; too much looks invasive. We constantly regulate gaze in conversation, often without noticing.

Past work on gaze-aware avatars often sat in VR, semi-immersive settings, or scripted dialogue. Today’s AI avatars are becoming combinations of LLM, speech recognition, speech synthesis, and 2D UI. Speech grows more natural, but gaze is often fake, random, or forever Mona Lisa — “seeming to look at everyone.”

The authors therefore ask something concrete: on an ordinary 2D screen, if an LLM-driven avatar can sense the user’s gaze in real time and adjust its own eyes, do user behavior and subjective experience change?

System design: move only the eyes — keep other factors from muddying the results

To isolate gaze awareness, the authors made restrained choices. Avatars had slight idle body movement and lip synchronization, but no facial expression and no extra gestures — so expression, motion, and emotional voice would not confound results.

Participants could choose a male or female avatar with matching synthetic voice. Same-gender preference was clear: 5 of 6 women chose the female avatar; 8 of 11 men chose the male. The authors note this preference may be culturally shaped and should not be simply generalized.

The task was also light: the avatar played quiz master with general-knowledge questions. That gave participants some cognitive load so they would not stare at the avatar as if in an “eye-contact experiment,” closer to everyday conversation.

Gaze-aware model: look, hold briefly, then look away

Gaze awareness here is not a deep model — it is a finite-state machine with three states: random gaze, mutual gaze, and gaze break.

When the system detects the user looking at the avatar for long enough, the avatar enters mutual gaze and establishes virtual eye contact. If mutual gaze lasts too long, it enters gaze break and looks away, mimicking how humans avoid continuous staring in conversation.

As Figure 2 shows, the model has several time thresholds: for example, user look toward the avatar must exceed 300 ms before pending; gaze loss must exceed 200 ms to confirm interruption; mutual gaze of 3600 ms triggers gaze break. The goal is not that the avatar always stares at the user, but a brief, responsive, not overly invasive gaze loop.

Real-time interaction: LLM dialogue, speech, and eye tracking must run together

Another value of the paper: this is not offline avatar playback or Wizard-of-Oz. The system wires together microphone, speech-to-text, OpenAI Realtime API, LLM response, text-to-speech, ChatVRM avatar, Tobii 4C eye tracker, and logging.

After the user speaks, the system recognizes speech in real time, lets the LLM generate a reply, synthesizes speech with TTS, and syncs avatar lip motion. Meanwhile, the Tobii tracker reports gaze coordinates at 90 Hz; a Flask server forwards them to the frontend to update the avatar’s gaze state.

As Figure 4 shows, the architecture’s point is not spectacle — it lets researchers log user gaze, avatar gaze, and dialogue events in unscripted real-time dialog. In other words, it offers a reusable experimental base for studying nonverbal behavior in AI avatars.

Study design: 17 people, each experiencing both versions

The user study used a within-subject design. Seventeen participants experienced a gaze-aware avatar and a non-gaze-aware baseline in counterbalanced order. In each round, the avatar asked general-knowledge questions as quiz master; participants answered aloud.

Objective measures came from eye tracking: proportion of time looking at the avatar, mean fixation-episode duration, episode rate, mutual gaze time, plus aversion rate and aversion duration when the avatar looked at the user and the user looked away.

Subjective measures used five 5-point Likert items: whether the avatar seemed interested in me, aware of my presence, natural in gaze, overall interaction satisfaction, and whether dialogue functioned normally. After both rounds, participants also directly compared which version felt more natural and which they would prefer to use in the future.

Core result: mutual gaze rose from 3% to 31% — yet users looked for shorter stretches

The strongest results are in the eye-tracking data.

Under gaze-aware conditions, the avatar’s looking-at-user time rose from baseline 0.084 to 0.553, p < .001; mutual gaze time rose from 0.033 to 0.306 — about 3% to 31%, p < .001; mutual gaze episode rate also rose significantly, p < .001.

Interestingly, total time users looked at the avatar slightly fell: baseline 0.524, gaze-aware 0.458, p = .042. Mean episode duration also shortened, from 0.919 s to 0.752 s, p = .022.

That is not failure — it looks like human conversation. When someone truly looks back, you do not keep staring; you glance away to regulate closeness and pressure. A gaze-aware avatar does not simply make users “look at it more” — it makes users regulate gaze as if facing an entity that can return a look.

More direct evidence: users look away more often — but each time is shorter

The authors further analyzed aversion: events where the user looks away while the avatar looks at them.

Under gaze-aware conditions, aversion rate was higher: baseline 0.706/s, gaze-aware 0.968/s, p = .003. Meanwhile, mean aversion duration was shorter: baseline 2.875 s, gaze-aware 0.721 s, p = .013.

That combination matters. Users are not fleeing the avatar entirely — they make more frequent, briefer gaze breaks during mutual gaze. That is closer to fine-tuning in human social interaction: look, look away, look back.

Subjective questionnaires were weaker: users may not know they were affected

If you only read the questionnaires, the story is milder. The gaze-aware avatar scored higher on most subjective items, but not significantly.

The largest difference was perceived naturalness: baseline mean 2.94, gaze-aware 3.41, p = .070 — only a trend. Overall satisfaction was slightly higher too (3.53 vs 3.82, p = .096). Functionality was slightly higher for baseline (4.41 vs 4.12).

Direct preference was only a mild lean. Eight people found gaze-aware more natural, six preferred baseline, three saw them as the same; future-use preference had the same split. Chi-square was not significant, p = .33.

As Figures 9/10 suggest, subjective differences were far less clear than eye-tracking. That is exactly the paper’s point: gaze awareness may first change low-level social reactions, not preferences users can clearly report.

What this means for AI avatar design

First, gaze is not a “visual detail” — it is an interaction mechanism. An avatar that responds to user gaze changes the user’s looking rhythm. Even if users do not notice, behavioral data can.

Second, 2D screens still deserve serious gaze design. Many assume mutual gaze is a VR, robot, or special-hardware problem. This paper shows that on ordinary 2D screens, with eye-tracking input and a sensible model, gaze-aware behavior can still trigger social-like regulation.

Third, lightweight models can work. This is not complex emotion inference — a finite-state machine of look, mutual gaze, look away. That already significantly changes mutual gaze and aversion. For products, gaze need not wait for full emotional intelligence.

Evidence strength: behavioral results are solid; subjective conclusions should stay conservative

The most persuasive part is objective eye-tracking. Mutual gaze time, avatar-to-user gaze, aversion rate, and related results show clear statistical differences, and they align with social-psychology ideas of intimacy balance, turn-taking, and gaze aversion.

But subjective questionnaires did not differ significantly — so experience-level claims like “looks more natural,” “more satisfying,” or “I’d use it more” cannot be locked in. The sample is only 17, in a well-controlled lab — not everyday office, customer service, teaching, or companion settings.

Also, the study deliberately removed facial expression, gesture, and emotional tone to isolate gaze awareness. That helps experimental control, but real avatar products are multimodal. Future work must ask whether gaze, expression, vocal emotion, and body posture amplify or interfere with each other.

The larger issue: AI that can “look at people” also needs boundaries

The paper ends on an important ethical point: gaze operates below conscious awareness. Gaze feedback can affect connection, comfort, and trust without users clearly noticing they are being influenced.

For AI products that is a double-edged sword. A virtual teacher who looks appropriately at students may feel more attentive and companionable; a health assistant that signals listening with eyes may reduce distance. The same mechanism can also manufacture over-dependence or manipulative intimacy.

So the question for gaze-aware avatars is not only “can we make it human-like,” but “in which scenes, at what strength, and how transparently should we use this social signal toward users.”

Key terms

Gaze Awareness: the avatar can sense whether the user is looking at it and adjust its own gaze behavior accordingly.

Mutual Gaze: user and avatar look at each other at the same time — virtual eye contact.

Gaze Aversion: actively looking away when the other party looks at you — a common way humans regulate closeness and pressure in conversation.

Mona Lisa Effect: figures in flat images that seem to look at the viewer from many angles, which makes true eye contact harder to design for 2D avatars.

References

Drewes, H., Dinc, A., & Alt, F. 2026. Investigating Mutual Gaze in Real-Time LLM-Driven Avatar Interaction on 2D Screens. Proceedings of the ACM on Human-Computer Interaction, 10(3), Article ETRA005. https://doi.org/10.1145/3806019