AI Trust Research | Explaining Reasoning Does Not Equal Being Trustworthy

Many AI products now like to show their “thinking.” Under the answer sits a stretch of text that looks like step-by-step derivation, and users naturally feel: the model did not answer offhand — it seems to have reasoned.

That is also where the problem sits. Fluent reasoning does not mean the reasoning is correct; a confident tone does not mean the evidence is enough. Once an AI rationale appears in the interface, it is no longer only an internal model capability — it becomes an interface cue that can shape judgment.

What makes this arXiv preprint valuable is that it takes apart the intuition that “a little more explanation is always good.” Across an online experiment, an eye-tracking study, and exploratory predictive models, the authors show: LLM reasoning text does change trust, but it does not automatically produce better decisions; the key is not showing more reasoning, but making reasoning more auditable.

01 Reasoning text is not a transparency window — it is a trust cue

The paper’s question is concrete: when an LLM shows its reasoning process directly to users, do people trust the information more, trust the system more, and adopt AI recommendations more readily? Further, are users actually “reading” the reasoning, or checking it against evidence?

The authors place this question in factual verification — a fact-checking task. The scene’s advantage is that answers can be judged right or wrong, and rationales can be designed as consistent or flawed. More importantly, users must decide: accept the AI’s answer, or go back and verify?

As Figure 1 shows, the paper has three parts: Study 1 is an online experiment manipulating how reasoning text is presented, its correctness, and certainty wording; Study 2 is a lab eye-tracking study of how users look at evidence, answers, and rationales; finally, gaze features feed exploratory user-state prediction.

The design’s through-line can be put in one sentence: a good rationale should not only “look reasonable” — it should make it easier for users to check the relationships among answer, evidence, uncertainty, and reasoning. The authors call this goal auditable trust calibration. They also carefully note that this is a design stance, not a calibration outcome directly measured in the paper.

02 Presentation format matters less; reasoning quality and certainty matter more

Study 1 recruited 68 English-speaking participants in a mixed design: rationale correctness was correct or incorrect; certainty cue was none, certain, or uncertain; presentation format was immediate, delayed, or on-demand expand. Each participant completed 6 fact-verification trials.

One control matters a lot: the LLM’s final binary answer was always factually correct; what varied was whether the rationale was consistent with that answer. So an incorrect rationale was not a wrong answer — it was “a correct answer followed by a flawed explanation.” That isolates the effect of reasoning text itself, but also limits how far conclusions can generalize.

Figure 2 shows the Study 1 interface. Users saw the question, LLM answer, reasoning process, and any certainty cue. In the on-demand condition, the rationale was hidden by default; users could expand or collapse it with a button.

The results are interesting: the paper found no reliable effect of presentation format. Immediate, delayed, and on-demand were close in mean trust in information, trust in the LLM system, and decision confidence. What acted more stably was rationale correctness and certainty framing.

Figure 3 shows that correct rationales, versus incorrect ones, brought higher information trust, system trust, decision confidence, and recommendation adoption. Certainty cues also raised trust and adoption; uncertainty cues lowered those ratings. In other words, users are not only affected by “whether reasoning is shown,” but by whether the reasoning seems sound and whether the tone sounds certain.

03 Bad reasoning makes people trust less — but that does not mean decisions get more accurate

Looking only at Study 1, it is easy to overclaim: correct explanations raise trust, incorrect ones lower it, so explanation helps calibrate trust. Study 2 pulls that judgment back a step.

Study 2 recruited 54 lab participants in a three-condition within-subjects design: no rationale, correct rationale, incorrect rationale. Each person completed 12 fact-verification trials, 4 per condition. For eye-tracking analysis, one participant’s gaze data quality was insufficient, so 53 participants’ data were used.

Figure 6 shows Study 2 self-report results. Incorrect rationales brought the highest cognitive load and the lowest scores on both trust measures; correct rationales had higher trust in information and trust in the LLM system. The manipulation check also held: participants could indeed perceive that correct rationales were more consistent with the answer.

But the key turn: in Study 2, rationale condition did not significantly affect decision accuracy, F = 1.40, p = .250; nor decision confidence, F = 1.44, p = .242. Accuracy was similar across conditions, roughly .77 to .82.

That clarifies the paper’s boundary. Rationales change how users evaluate information and the system, and how effortful processing feels — but in this task, they do not show that “displaying reasoning makes people decide better.” That is the most worth-keeping counterintuitive point.

04 Eye-tracking shows users inspect reasoning — but that is not yet “trust calibration”

Study 2’s eye-tracking moves the paper from “how users rate” to “how users look.” The authors divided the interface into four AOIs: context (question and supporting evidence); answer (LLM final answer); rationale; and response/rating (user answering and rating areas).

Figure 7 shows that incorrect rationales, versus correct ones, brought more context-oriented scanning — higher fixation counts, saccade counts, and saccade lengths. When reasoning had holes, users were more likely to scan back over the evidence region.

A subtler detail: incorrect rationales did not make users dwell longer on the rationale region — fixation counts and durations there did not reliably change with correctness; but incorrect rationales brought larger fixation-level pupil diameter in the rationale AOI. The authors read this as process evidence of higher processing effort or cognitive load, not as a direct readout of trust.

That matters. Gaze can show how attention is allocated and hint that people may be checking, confused, or working harder. Gaze is not a “trust sensor.” The paper’s careful claim: these gaze patterns relate to audit-like inspection, but do not prove successful trust calibration.

05 Gaze can predict some states — but it is not real-time mind reading

Finally the paper runs post-hoc predictive modeling. The authors combine Study 2 AOI-level gaze features with some background variables and use leave-one-subject-out cross-validation to predict trial-level self-reports or decision states such as trust, cognitive load, decision accuracy, and decision confidence.

Gaze features do carry some retrospective signal. For example, with background variables, trust in information reached up to .67 accuracy and .72 F1 in pooled analysis; trust in the LLM system up to .70/.70; cognitive load up to .72/.79; decision confidence up to .72/.82 in the correct condition. Decision accuracy scores were highest: .80 accuracy and .88 F1 in the pooled setting, and .84 accuracy / .91 F1 on incorrect-rationale trials.

Figure 8 gives exploratory SHAP feature-attribution. Across targets, important features spread across context, answer, rationale, and response AOIs rather than concentrating in one region — the models use a set of task-related visual behavior patterns.

The authors do not package this as deployable real-time trust detection. Three reasons: labels come from post-trial self-reports (after-the-fact states); decision accuracy already has a high base rate, so scores need that context; and the paper admits mild label-definition leakage from the median threshold. Safer phrasing: in this fact-verification task, gaze carries limited, situation-specific behavioral signal — far from general user-state recognition.

06 The product lesson is not “show more reasoning” — it is “make reasoning checkable”

The design implication should not be simplified to “show chain-of-thought.” On the contrary, the authors oppose maximal disclosure and persuasive narrative. If reasoning text is only longer, more human-like, and more certain, it may increase persuasiveness without increasing verifiability.

Four better directions. First, secure rationale quality before animation, delay, or on-demand reveal. Study 1 found no reliable presentation-format effect — reveal mechanics cannot replace the reliability of the reasoning itself.

Second, treat the rationale as an audit trace, not a persuasive narrative. Steps should be separable, assumptions visible, and key claims mappable to evidence. Third, certainty cues should align with evidence, answer, and rationale quality — do not wrap uncertain reasoning in a certain tone. Fourth, if future adaptive support uses eye tracking or other sensing, it must be transparent, optional, privacy-preserving, and serve user autonomy rather than cultivate dependence.

The sentence worth taking away: AI reasoning text is not better when there is more of it, or when it looks more like thinking; it must make it easier for users to know where to trust, where to question, and where to verify.

07 The paper’s boundaries matter more than its conclusions alone

The limitations are stated clearly and should shape how we read it. First, the task is short binary fact verification; rationales were preconstructed and checked by the authors — not naturally generated reasoning in open, multi-turn, high-stakes settings.

Second, in both studies the LLM answer was always correct. So the work cannot answer whether users can reject wrong AI advice. Lower trust after incorrect rationales might mean users found reasoning holes — or unnecessary doubt next to a correct answer. Future work needs to cross answer correctness with rationale correctness to separate productive auditing from miscalibrated rejection.

Third, Study 1’s item pool had only 6 counterbalanced items, so condition effects partly confound with item difficulty; Study 2’s sample was younger and more female. Fourth, the on-demand condition did not log whether participants expanded the rationale or for how long, so “no presentation-format effect” should not be overread.

These limits do not weaken the paper’s value — they make it more usable. It does not promise a universal answer; it reminds us: once LLM rationales become interface, we cannot only ask whether they raise trust — we must ask whether they deserve trust, can be checked, and might push users toward false certainty.

Paper information

Original title: "When LLM Rationales Become User-Facing: Effects on Trust Perception, Decision-Making, and Gaze Behaviors". Authors include Xin Sun, Ting Pan, Yajing Wang, Shu Wei, Jos A. Bosch, Isao Echizen, Abdallah El Ali, and Saku Sugawara. Affiliations include National Institute of Informatics, University of Amsterdam, Yale School of Medicine, University of Tokyo, CWI, and Utrecht University. Paper link: https://arxiv.org/abs/2606.25489

Key terms

LLM rationale: here, the reasoning text shown to people in the user interface — not identical to the model’s true internal reasoning. This paper cares about how it, as an interface element, affects trust, adoption, and evidence-checking behavior.

Auditable trust calibration: the authors’ design goal — that a rationale should help users check whether the answer has evidence support, whether certainty is warranted, and whether further verification is needed. The paper does not directly prove successful calibration; it studies trust, adoption, and gaze processes related to that goal.

AOI: Area of Interest — predefined interface regions in eye-tracking research. This paper splits the interface into context, answer, rationale, and response/rating to analyze where attention goes.

References

Sun, X., Pan, T., Wang, Y., Wei, S., Bosch, J. A., Echizen, I., El Ali, A., & Sugawara, S. (2026). "When LLM Rationales Become User-Facing: Effects on Trust Perception, Decision-Making, and Gaze Behaviors". arXiv:2606.25489v1.