CHI 2026 | Everyone Has an LLM — Why Do Technical People Still Win?

The most seductive promise of putting LLMs into programming courses is lowering the barrier: you need not know Python, or be fluent in data analysis — ask well enough, and the model will fill in what you lack.

Real classrooms are messier. Some students treat Gemini as an on-call partner: plan, generate, then check. Others type "fix it" again and again and spin on the same KeyError. Same tool, similar usage frequency — outcomes diverge.

This CHI 2026 study followed 36 graduate students through a semester of data-science homework. Its conclusion is not especially flattering: LLMs can temporarily shrink gaps on timed tasks, but once time opens up, technical experience still significantly predicts grades. Self-rated LLM familiarity and communication skill did not show the same explanatory power.

The paper does not stop at the common sense that "experts beat novices." It keeps asking when the two groups call AI, how much context they give it, how they decompose tasks, and whether they understand and verify outputs. That makes the takeaway concrete: AI literacy cannot teach prompts alone — it has to cover a full problem-solving repertoire.

01 Using AI is common; having spare capacity to use it well is hard

The study took place in a 2025 spring graduate course, Data Science for Product Managers, at a U.S. R1 university. Forty-one students completed diagnostic homework HW0; 36 persisted through HW4. Backgrounds ranged widely — from computer science and physics to finance, journalism, and music — exactly the spread needed to test whether LLMs truly flatten starting points.

The course centered on four kinds of Python data-analysis tasks: cleaning, exploratory analysis, lightweight machine learning, and data storytelling. Students used built-in Gemini in Google Colab; external GenAI tools were in principle disallowed. The team collected notebooks, grades, surveys, screen recordings, and interaction logs; after quality filtering, log analysis covered 28 students and 7,315 events.

Authors did not collapse all experience into a vague "fluency." They separated three self-rated measures: technical experience, LLM experience, and communication experience. As the figure shows, the sample's median technical experience was only 1.97/5, LLM experience 3.67/5, and communication experience 4/5. The awkward combination: most people felt they could communicate and were not strangers to LLMs, yet technical foundations differed sharply.

That split matters. If "uses ChatGPT often" is treated as AI ability, we easily overvalue familiarity with a tool interface. Later results show that everyday usage experience did not automatically transfer to open-ended data-analysis work.

02 Gaps vanish on timed tasks — and return when time opens up

The design embeds a contrast. HW0 was an in-class diagnostic, about 15 minutes per item; half the students used GenAI on some tasks, half did not. Without LLMs, students with different technical experience differed significantly in scores (Mann–Whitney U, p=0.027). Once LLM use was allowed, that difference was no longer significant.

If the story stopped there, one might declare that LLMs are closing the ability gap. But HW1–HW4 were biweekly assignments closer to real learning: students had more time and free use of Colab Gemini. Authors analyzed grades with linear mixed-effects models, treating student and homework as random intercepts. Results: only technical experience significantly predicted higher grades, β=6.09, p=.041. LLM experience (p=.266) and communication experience (p=.381) did not reach significance.

Bring time scale in, and the two results align. Under time pressure, the model can quickly supply code and help novices clear immediate operational hurdles. Once tasks lengthen, the hard part shifts from "can I generate a snippet" to "what should I do, what does the model need to know, is the answer reliable, where next." Those judgments still depend on technical knowledge.

Do not rush to write "technical experience decides everything." The sample is only 36; communication scores cluster high, so variance available to the model is limited; experience measures are self-rated. The paper offers significant associations in a classroom setting, not a pure causal effect of technical experience on grades.

03 Novices and veterans both use AI — the difference is when they step in

To see the process behind grades, authors reorganized logs into problem-solving episodes, then into four stages: intent, input, understand, and assess. The 7,315 log events mapped to 1,483 steps; 920 were labeled as using AI or missing a suitable AI opportunity. Auto-labeling used an LLM with 10% spot checks; overall accuracy was 75%, so these distributions are better for finding patterns than as error-free behavioral measurement.

Figure 4 shows AI behavior against work stages: planning mainly at intent formation, code generation at input, explanation at understand, evaluation at assess. More striking: students often did not call AI at explain and assess; at input, clear instructions and sufficient context were missed 46% of the time.

Call counts did not separate groups. Veterans and novices both used AI in about 36% of steps, but veterans more often planned with it before problems arose and more often handled difficulties themselves. Facing challenges, veterans requested explanations 75% of the time versus 57% for novices; when novices used AI to improve or plan, they were already in difficulty 64% of the time, versus 43% for veterans.

In the logs, the two groups' work styles already diverge. Veterans treat AI as a cognitive resource: judge the task first, then decide where to borrow leverage. Novices more easily treat it as a panic button — they ask only after errors. Both groups appear to be "using AI," but one is allocating work; the other is chasing errors.

Code-generation success continues the line: veterans using AI to generate code succeeded on 90% of steps, novices on 79%. In high-scoring veterans' prompts, quality features such as clear instructions, task decomposition, and context were about twice as common as in other groups.

Here "prompt craft" returns to its proper place. A high-quality prompt is not a magic sentence — it is a compressed expression of goals, constraints, and context after the user already understands the problem. Without that understanding, prompt templates quickly degrade into a more polite, more complete "please fix this for me."

04 One demo can improve asking — it rarely teaches judgment

Between HW0 and later homework, the instructor ran a 90-minute live-coding demonstration showing how to use Gemini to plan, explain concepts, and generate code, and which contexts different Colab entry points can access. Later homework gave students more time and ordinary data-science instruction.

After training, episodes more often reached the assess stage instead of stopping at intent or input. Figure 7 makes the shift visible: in the pretest, many flows ended before forming a valid intent; later logs show more students advancing through understand and assess.

In concrete behaviors, clear instructions, providing context, and task decomposition all improved; appropriate use of prompt-quality-related behaviors rose about 30% on average. But critique output, explaining code, or explaining errors — assessment behaviors — still had appropriate-use rates below 50% after training.

Lightweight teaching hits a wall here. Watching a teacher write prompts once, students can quickly mimic surface moves. Facing a long explanation or a mixed true/false code answer — whether they pause to read, can set verification criteria, when to abandon the current strategy — needs repeated practice and feedback.

Note an important confound: the paper does not isolate the 90-minute demo's effect. Pre–post change also includes more homework time, regular course learning, and ongoing practice, so it is only a preliminary signal; one cannot attribute the full 30% change to that demo alone.

05 The model did not see it — students assumed it knew everything

The paper's most vivid stretch is how the same KeyError unfolds for veterans and novices. Veterans first check actual column names; finding the model used the wrong field, they put the correct names into the prompt or edit the code directly. Novices repeatedly tell the model "do not use columns that do not exist" without providing fields that actually exist in the data — and the error repeats.

What stalls here is not one prompt, but a distorted mental model of the system. Colab's code-generation entry and side chat do not see the same context, and the model may not access column names of an imported dataframe. Some students transfer ChatGPT/Perplexity habits and assume Gemini should "already see what is on screen."

So AI literacy needs at least two maps: a task space — what work can be handed to AI — and a context space — which files, variables, code, and history the current entry can actually access. Buttons change as tools update; the ability to calibrate these two maps does not age out.

An "Explain error" button also need not improve learning. It lowers the cost of requesting explanation, yet may return a long, hard-to-digest text. For cognitively overloaded novices, more information is not easier assessment — sometimes it only buries the error in a longer answer.

06 Prompts are the surface; courses must complete the full loop

Authors split AI-use ability into four dimensions: conceptual knowledge answers "what can AI do and see"; procedural knowledge answers "how to give clear goals, context, and completable subtasks"; metacognitive knowledge answers "when to call AI and when to change strategy"; dispositional capacity concerns willingness to read long outputs, tolerate trial and error, and keep verifying.

Courses might set prompt templates aside first and have students write success criteria before generating — which columns to keep, what constraints the output must meet, how to judge whether a chart answers the question. After generation, require verification via data checks, local runs, or result comparison.

For tool design, one-shot long answers can become stepwise diagnosis. When the system sees a user repeatedly submit something like "fix it" and hit the same KeyError in a short window, it need not keep generating full code — it can first prompt running df.columns, checking real fields, then proceed. That scaffold does not finish the task for the user; it makes verification executable.

The judgment I find most worth taking from this paper: plugging in an LLM only solves "is help available," not "who knows how to get help." If classrooms only count usage and products only optimize call entry points, both miss the abilities that actually move outcomes.

Still, keep the extrapolation range tight. The study comes from one graduate course, a small sample, and tasks that are not especially LLM-hard for current models; Python, data science, and tool experience also correlate. Colab UI and underlying models keep changing, and custom browser logs may miscount autocomplete as code edits. Larger samples, performance-based pretests, and controlled experiments are still needed to separate technical knowledge, AI-use experience, time pressure, and instructional intervention.

Paper information

Original title: Not Everyone Wins with LLMs: Behavioral Patterns and Pedagogical Implications for AI Literacy in Programmatic Data Science

Authors: Qianou Ma, Kenneth R Koedinger, Tongshuang Wu

Affiliation: Carnegie Mellon University Human-Computer Interaction Institute

Venue: CHI 2026, Barcelona, Spain

Paper link: https://doi.org/10.1145/3772318.3791283

Key terms

AI literacy: Beyond tool familiarity — a workflow capacity of conceptual, procedural, metacognitive, and dispositional abilities.

Task space: Which tasks current AI can take on — planning, explaining, generating, local edits, assessment.

Context space: Which code, variables, data, files, and dialogue history the current AI entry can actually access.

References

Ma, Q., Koedinger, K. R., & Wu, T. (2026). Not Everyone Wins with LLMs: Behavioral Patterns and Pedagogical Implications for AI Literacy in Programmatic Data Science. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. ACM. https://doi.org/10.1145/3772318.3791283