Multimedia Systems 2026 | Can a Large Model Predict Where VR Users Look 250 ms Ahead?

If a VR system can know where you will look next, it can compute that small patch of the frame first.

That sounds like gaze prediction, but doing it well involves a whole body–scene relationship. In six-degree-of-freedom space people move and turn their heads, and also rotate their eyes in head coordinates; salient regions such as exhibits, roads, and mountain peaks pull attention in different directions. Predicting only a fixation point on a screen does not describe full viewing behavior.

Published in Multimedia Systems, GBP-LLM predicts head translation, head rotation, and eye direction as three related tasks together. It uses a frozen large language model as a sequence-fusion backbone, but what actually decides success is not having the model generate language — it is getting the geometry right first, then sending scene cues into the same representation space.

One counterintuitive result appeared in the experiments: on this task, frozen GPT2-xl beat adding LoRA fine-tuning; explicit spatial geometry also mattered more than a fancier language-model name.

01 Predicting the next glance cannot only watch the eyes

In ordinary 2D interfaces, gaze location is often expressed as on-screen coordinates. Six-DoF VR adds a layer of spatial motion: users can move forward/back and left/right, and also pitch, yaw, and roll. Which ray into the world they ultimately look along is jointly decided by head position, head orientation, and eye direction relative to the head.

So the paper does not shrink the task to one gaze-point coordinate. The model jointly predicts future head translation, three Euler angles, and 3D eye direction, and additionally computes combined gaze error in world coordinates. That last metric matters especially: head and eyes may each look only slightly off, yet the true combined gaze can still be farther wrong.

Predictions can serve gaze-aware interaction and pre-rendering. If the system can judge gaze landing early, it may prioritize rendering regions about to enter the center of attention. This paper, however, evaluates trajectory error and inference latency; it does not directly test whether image quality, interaction experience, or cybersickness improve as a result.

02 Eighteen people and three scenes make up the training data

The study recruited 18 participants, ages 20–24, mean 22.67, 12 men and 6 women. Wearing HTC Vive Pro Eye, they freely viewed three virtual environments — museum, campus, and forest — with no assigned task; scene order was randomized, and each experience lasted about 4 minutes.

As Figure 1 shows, the three environments differ clearly in visual structure. Museum attention targets concentrate on wall exhibits; campus includes roads and building façades; forest has more dispersed terrain, trees, and rocks. That difference is exactly why scene context needs to enter the model.

Final data contain 54 files — records of 18 people each experiencing 3 scenes. Head, eye, per-frame saliency maps, and participant/scene IDs are aligned at 60 Hz. Authors use 75%-overlap sliding windows, yielding 51,624 sample windows, and report averages with 10-fold cross-validation.

Modality synchronization is the most solid part of this dataset. Saliency maps feed directly into the Unity render pipeline rather than processing only a few keyframes. But participants are young and few; all three scenes were built by the research team; viewing had no concrete task. It suits validating short-horizon trajectory prediction and does not yet represent gaze strategies in games, collaboration, training, or high-pressure interaction.

03 Triangle Encoder first puts three motions back into one space

Raw data are not naturally comparable. Head position lives in world coordinates; head rotation is pitch, yaw, and roll; eye direction sits in head coordinates. Concatenating these numbers lets the model see variables without necessarily knowing how they jointly point at a target in space.

Triangle Encoder builds a spatial triangle for each frame. The three vertices are head position, the intersection of head-forward with a canonical view plane, and the intersection of the eye ray with the same plane. Head position, orientation, and eye deflection are thus unified into a 9-D geometric representation, then encoded over time by a linear layer and a two-layer Transformer.

Figure 2 shows the full architecture. Geometric encoding on the left becomes continuous soft tokens via a Projector; Prompt Builder on the right writes task description, participant and scene IDs, and salient regions as structured prompts. The two streams fuse through self-attention in a frozen LLM; three decode branches finally output head position, head orientation, and eye direction.

Ablation shows the triangle is not decoration. Removing Triangle Encoder raised mean head-position error from 0.120 m to 0.167 m and combined angular error from 3.064° to 3.835° — the largest effect among deleted components. Contribution order is therefore clear: write cross-modal relations into geometric structure first, then talk about unified representation.

04 The prompt here is not just one sentence to the model

Prompt Builder has three parts. System Prompt states the temporal prediction task and I/O format; Context Prompt provides scene ID and participant ID; Saliency Prompt clusters three high-saliency centers from the saliency map, then uses head pose to lift 2D coordinates onto the 3D view plane.

This prompt is not translating a string of coordinates into natural language and asking a chatbot to guess. The model receives both text tokens and soft tokens projected from motion sequences; the LLM fuses heterogeneous sequences. Backbone parameters stay fully frozen; training updates only peripheral encoding, projection, and decoding modules.

Authors compared Qwen2.5-Math-1.5B, DeepSeek-R1-Distill-Qwen-1.5B, and GPT2-xl. DeepSeek had the lowest head-orientation error; GPT2-xl was better on the other three metrics and was chosen as the default backbone. A newer-looking model name does not mean it suits this numerical sequence better; representation and training stability may matter more.

05 Error is lower — but the speed bill must be counted too

The study compared GBP-LLM with Informer, TimesNet, Dgaze, GazeTransformer, and FoANet. Figure 3 lists errors for future frames 1 through 15. At 60 Hz, 15 frames correspond to about 250 ms. Farther horizons worsen all models, but GBP-LLM kept the lowest error on head position, head orientation, eye angle, and combined angle throughout.

Averaged over 15 prediction horizons, GBP-LLM’s head-position error was 0.120 m, head-orientation error 2.182°, eye-direction error 1.499°, and combined gaze error 3.064°. Qualitative examples in Figure 5 also show that in museum, campus, and forest, predicted points usually sit near true fixations and near high-saliency regions.

Authors also rechecked on the public FixationNet dataset. That data lacks six-DoF head translation, so only angular metrics are compared. In Figure 6, GBP-LLM’s mean head-orientation, eye, and combined angular errors were 1.610°, 0.812°, and 1.929° — still below the same baselines. The public dataset blocks the objection that results work only on self-collected scenes, but still does not answer cross-device, cross-age, or cross-task generalization.

Beyond accuracy, the speed gap cannot be omitted. On NVIDIA H100, GBP-LLM’s mean batch inference time was 1149.952 ms; light baselines were about 19.968–45.056 ms. Converted by the paper’s batch size, per-sample time is about 8.98 ms, below the shortest prediction window, so authors argue it meets the experimental time granularity. That is not end-to-end latency of a full online system — saliency computation, data transfer, and render scheduling still need separate accounting.

06 Frozen beats LoRA — a result, and a reminder

Authors tried multiple LoRA settings; every fine-tuned version underperformed the fully frozen backbone. In Figure 7, LoRA curves rise faster with prediction distance; head-position and combined-gaze errors both exceed the frozen version. At least under current data scale, hyperparameters, and this numerical prediction task, further changing large-model parameters can damage existing representations rather than bring better domain adaptation.

This cannot be written as a general claim that LoRA is unfit for temporal prediction. The paper tested limited configurations; changing backbone, data volume, or training strategy could change results. In these experiments, GBP-LLM’s gains mainly come from geometric encoding, saliency prompting, and multi-task objectives; freezing the backbone preserved a stable shared representation space.

One more place in the evidence chain needs caution. The paper states 75%-overlap sliding windows and 10-fold CV, but does not say whether each fold isolates by participant or original file. 51,624 windows are not 51,624 mutually independent observations. If adjacent highly overlapping windows land in both train and test folds, error may be optimistic. Consistent public-dataset results ease that worry but do not replace a clear grouping strategy.

The paper itself lists three limitations: 2D saliency models may not fit VR’s non-Euclidean space; sound and other environmental cues are not yet in the model; LLM inference cost is unfavorable for latency-sensitive tasks such as gaze-point pre-rendering.

This paper orders the engineering steps clearly: unify geometry first, then bring in semantics, and only then decide whether to touch backbone parameters. It does not prove large models already understand human attention. For VR prediction systems, accurate prediction is only the first gate; still delivering timely results for new users, new devices, and real tasks is what gives a chance to enter render and interaction pipelines.

Paper information

Original title: GBP-LLM: Gaze Behavior Prediction in 6DoF VR via Large Language Models

Authors: Ding Ding, Wenjie Zhang, Zicheng Liu, Chang Qi, Zheyu Cao, Jinghui Zhang

Affiliation: School of Computer Science and Engineering, Southeast University

Journal: Multimedia Systems, 2026, Volume 32, Article 392

Paper link: https://doi.org/10.1007/s00530-026-02456-x

Key terms

6DoF: Six degrees of freedom, including translation along three spatial axes and rotation about three axes. This paper jointly predicts head position, head orientation, and eye direction.

Visual saliency: Regions in an image more likely to attract attention. Authors lift saliency centers onto the 3D view plane to provide scene cues for future gaze.

Soft token: An input representation formed by projecting continuous motion features directly into the large-model embedding space, without going through a natural-language vocabulary.

References

Ding, D., Zhang, W., Liu, Z., Qi, C., Cao, Z., & Zhang, J. (2026). GBP-LLM: Gaze behavior prediction in 6DoF VR via large language models. Multimedia Systems, 32, 392. https://doi.org/10.1007/s00530-026-02456-x