CHI 2026 | Emotion Annotation Need Not Label Everything — Catch Experience Turning Points

Anyone working in affective computing knows that the most expensive resource is often not the model, but the labels.

If researchers want to know how frustration, immersion, or arousal change during a game, training session, or learning episode, the most reliable method is usually to have the person rewatch the full experience and continuously annotate their own emotion curve. The problem is that this is slow and exhausting. Annotation often takes nearly as long as the experience itself; over time, fatigue, memory bias, and halfhearted labeling creep in.

PREFAB, proposed in this CHI 2026 paper, is not trying to make people annotate more finely. It asks a more practical question: can we annotate less without losing the critical structure of emotional change?

📌 Paper information

Original title: PREFAB: PREFerence-based Affective Modeling for Low-Budget Self-Annotation

Authors and affiliations: JaeYoung Moon, Youjin Choi, Yucheon Park, David Melhart, Georgios N. Yannakakis, Kyung-Joong Kim. Authors are affiliated with Gwangju Institute of Science and Technology, University of Southern Denmark, University of Malta, and related institutions.

Venue: CHI 2026, Barcelona, Spain.

Paper link: https://doi.org/10.1145/3772318.3791702

The problem is not that emotion is hard to measure, but that full annotation is too costly

Affective computing needs to capture internal states such as frustration, immersion, and arousal. These states change over time, so researchers often need a continuous curve rather than a single label.

Traditional retrospective self-annotation asks participants to rewatch the full experience after a task and annotate emotional state along a timeline. Its strength is detail; its cost is obvious: participants must re-experience the material, attention drops, memory blurs, and annotation quality can degrade with fatigue.

Some methods already try to annotate only "the right moments," for example by using physiological signals to find segments worth labeling. Those approaches often require invasive sensors and handle unlabeled segments poorly.

PREFAB takes a different path. It does not ask users to draw the whole curve from start to finish. Instead, a model first estimates emotional trends, finds the most critical inflection regions, has users annotate only those segments, and interpolates the rest.

Core assumption: people remember not average experience, but peaks and endings

PREFAB rests on two theoretical handles. The first is the peak-end rule: when people recall an experience, they are often more influenced by the most intense moments and the ending than by a simple average of the whole. The second is ordinal representation: asking people which of two segments felt higher may fit how they judge experiential change better than demanding absolute numeric values.

So PREFAB does not directly predict an absolute arousal score at a given second. It learns relative preferences between segments: higher or lower than another. Through that preference learning, the model first reconstructs an expected emotion trajectory, then finds peaks and turning regions in it.

That is what "preference-based" means in the title. The authors care less about single-point emotion values than about the relative order of emotional change.

Why not ordinary regression? Emotion annotation is not a precise thermometer

The paper compares PREFAB with cardinal regression. Ordinary regression tries to predict absolute arousal values and looks more direct, but it assumes self-annotation can be treated as stable numbers. The problem is that self emotion labels often have individual scale differences: one person's 0.7 is not necessarily another's 0.7.

The authors therefore use ordinal cross entropy and distinguish it from ordinary binary cross entropy. In short, the model does not only decide which of two segments is higher; it also uses finer relative order structure so the learning objective stays closer to the nature of emotional change.

As Figure 2 shows, OCE emphasizes ordered intervals more than BCE, which compresses the relation into a single binary boundary. That matters for emotion trajectories, because researchers care about where things rise, fall, and turn.

How does the model know where a turn might be?

PREFAB's training does not rely only on local segment comparisons. The authors also add an auxiliary task that clusters arousal time series from the AGAIN dataset with Dynamic Time Warping, yielding coarser emotion-change patterns.

The auxiliary task addresses the shortsightedness of local comparison alone. Change between adjacent segments can tell local trend, but not necessarily the long-term structure of an experience. DTW clustering helps the model learn more global trajectory shapes.

As Figure 3 shows, the authors first cluster time series, then use cluster results as auxiliary labels. The design is a bit like giving the model a rough map: local comparison tells the slope underfoot; the auxiliary task tells which kind of terrain the whole stretch belongs to.

PREFAB's architecture: compare two segments, do not look at one point in isolation

On the input side, PREFAB uses two consecutive segments, each containing 12 image frames, game log features, and participant biography information. The segments are one second apart, to learn the direction of emotional change.

Architecturally, images go through a CNN, game logs and biography through an MLP, then into a transformer encoder. Biography modulates feature expression via FiLM. The model jointly handles a main task and an auxiliary classification task: the main task learns inter-segment preference; the auxiliary task learns coarse emotion patterns.

As Figure 4 shows, the system resembles a Siamese network: it does not judge a time point in isolation, but compares two temporal segments. That structure matches the paper's problem, because PREFAB cares about "change," not static labels.

Technical evaluation: PREFAB catches turns better, but is far from perfect

The authors first run a technical evaluation comparing PREFAB, random sampling, uniform sampling, rule-based event-driven sampling, and cardinal regression. Metrics include region-level F1 and ΔTE.

F1 measures how well predicted turning regions overlap ground-truth turning regions. ΔTE measures the sampling region's gap from ground truth in time efficiency. That metric matters, because F1 alone can deceive: a method that covers almost the whole timeline may look highly overlapping while losing the point of "annotating less."

As Figure 5 shows, PREFAB performs best overall across 9 games. The paper reports that PREFAB significantly outperforms all baselines on F1; ANOVA shows method differences are significant, p < .001. On ΔTE, PREFAB averages 0.138 ± 0.033, lower than random at 0.154 ± 0.004, regression at 0.197 ± 0.041, rule-based at 0.241 ± 0.051, and uniform at 0.310 ± 0.032.

Still, the result should be bounded. The authors note that even as the best method, PREFAB's F1 remains below 0.7, so turning detection still has clear room to improve. It has not solved emotion annotation; it has pushed "annotate less without losing structure" one step forward.

User study: labeling less does not always save time — preview is the key variable

The second study is a 25-participant user study. The authors compare three conditions: Baseline (full annotation); PREFAB without preview; PREFAB with preview. Preview here means brief contextual cues that help users understand what is happening around the segment they must annotate.

The design is smart, because giving users only a short segment can leave them without cause and effect. Annotating less lowers load, but insufficient context can hurt confidence and quality. Preview pads the gap between "saving time" and "keeping grounds for judgment."

As Figure 6 shows, PREFAB with preview significantly reduced mental and physical load while increasing confidence. Participants also more often chose PREFAB with preview as the best method.

Quality did not clearly drop — but that claim needs a condition

The easiest misreading is: PREFAB can annotate less, so quality must stay the same. The paper is more cautious.

The authors find that PREFAB with preview shows no significant difference from full annotation on perceived accuracy and willingness to correct; PREFAB without preview has lower quality perception. So annotating less is not magic. Users need enough context to believe their labels are accurate.

On quantitative consistency, PREFAB's reconstructed curves reach moderate absolute agreement with full annotation, CCC = 0.67; Spearman ρ = 0.69, indicating relatively high ordinal similarity; DTW similarity = 0.82, indicating strong overall shape similarity.

As Figure 7 shows, preview is critical for perceived result quality. The version of PREFAB that actually holds is not "annotate only short segments," but "annotate only critical segments, with enough context."

Where this paper is truly valuable

I think PREFAB's value is not only lower annotation cost. It changes a default assumption: we need not always treat continuous emotion curves as point-by-point sampling problems. For many experiential tasks, the structure of emotional change may matter more than every second's absolute value.

That has implications for HCI, game research, educational technology, and immersive experience evaluation. In many settings, researchers really want to know: where frustration appears, where immersion begins, where experience turns. If a method can stably catch those change points, lower-budget annotation may still be useful enough.

It also reminds us that low-budget annotation should not only chase asking users fewer questions. The real question is whether, after asking less, we still keep the structure needed to explain experience. PREFAB's answer is: try, but model, theory, and context design must support it together.

Limits: do not treat it as a universal "save annotation" button

This work has several boundaries. First, it uses the AGAIN dataset and game experience scenes; results may not transfer directly to medicine, education, driving, or VR training. Emotion turns in different tasks may have different timescales.

Second, PREFAB depends on the model first estimating turning regions. If the model misses a critical turn, users never get to annotate that stretch. Risk concentrates here: the method saves human attention and also hands part of the selection power to the model.

Third, preview raises confidence and quality perception, but can also add time. The paper notes that when preview-watching time is included, total annotation time sometimes reaches 136% of full annotation. PREFAB more reliably eases load; time savings are conditional.

Three judgments worth taking away

First, continuous experience annotation need not always cover every point. If the research question concerns change structure, catching turning points may be more effective than spreading effort evenly across the whole timeline.

Second, preference learning fits subjective experience. People's numeric emotion scales are unstable, but relative comparison is often more natural. PREFAB turns that into model design.

Third, saving annotation cannot mean saving context. Users can label less and reduce load, but without preview, confidence and quality perception drop. Low budget is not cutting information away; it is putting attention where judgment matters most.

Key terms

🧭 Self-Annotation: Participants label internal states from their own experience — arousal, frustration, immersion, and similar constructs.

📈 Inflection Region: Segments in an emotion trajectory where change is clear and may shape the remembered structure of experience.

🔁 Preference Learning: The model learns relative relations between two segments rather than directly predicting an absolute value.

🎞️ Preview: Contextual preview that lets annotators see brief surrounding context before labeling a critical segment, supporting more confident judgment.

References

Moon, J., Choi, Y., Park, Y., Melhart, D., Yannakakis, G. N., & Kim, K.-J. 2026. PREFAB: PREFerence-based Affective Modeling for Low-Budget Self-Annotation. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. ACM. https://doi.org/10.1145/3772318.3791702