CHI 2026 | Coding Agents Save Effort — Why Most Participants Still Prefer Copilot

A tool that can install dependencies, edit files, run tests, and help more people finish tasks correctly should, in theory, be more popular. Yet after this programming experiment, when asked whether they would prioritize OpenHands over GitHub Copilot in the future, 60% of participants said no.

That is not because the Agent failed to help. In the study, correct task completion rose from 25% under Copilot to 60% under Agent; among successfully completed tasks, estimated human effort time fell from 25.1 minutes to 12.5. More tasks done right — yet people willing to prioritize it still did not reach half.

This CHI 2026 paper compared how 20 participants used both tools. Once execution steps are handed to AI, how do developers know what it did, and how do they decide when to stop? Fewer commands typed does save effort; it does not necessarily bring a stronger sense of control.

01 Compare who executes — not only tool names

The authors place the two ways of working on a spectrum of automation. On the left of Figure 1, Copilot helps complete code and answer questions while people still perform many operations; on the right, an Agent can take a task, read files, modify code, run programs, and keep adjusting from results. The difference is who takes the next steps.

The experiment ran in April–May 2025, comparing GitHub Copilot with OpenHands, the latter using Claude Sonnet 3.7. That describes the configuration and workflow at the time — not a hard boundary for today’s products. The appendix also lists different Copilot modes; it should not be read as autocomplete alone.

Twenty participants were recruited through a university mailing list — students who regularly used Copilot and had Python experience; none reported prior use of a coding Agent. Sixty percent self-reported 3–5 years of professional programming experience; 95% rated their Python level intermediate or advanced. Familiarity with one side and novelty of the other will shape later judgments.

Figure 2 shows the arrangement: each person used both tools for about 40 minutes each, with order randomized. Tasks fell into three types — gather and analyze networked data, add a feature to an existing repository, and fix bugs — with two problems prepared per type; the same participant faced same-type tasks under both conditions.

These problems did not stop at generating a small function. One required adding a reversible dark mode to matplotlib; another required collecting web data and analyzing it. The study judged complete solution from final code and tests — not treating lines of generated code as productivity.

02 People worked half as much — that does not mean the task was twice as fast

Figure 3 shows task correctness on the left and user effort time on successful completions on the right. Correctness from 25% to 60% is an increase of 35 percentage points, not a relative lift of 35%. The authors report the difference as significant, p = 0.02; but the baseline is not “no AI at all” — it is already human–AI collaboration with Copilot.

The time numbers need careful reading of the metric. Under Copilot, timing ran from task start to finish; under Agent, interaction logs estimated time users spent reviewing results and composing the next instruction — periods of autonomous Agent running were not counted. The figures were 25.1 and 12.5 minutes respectively, with p = 0.01 for the difference.

If Agent time spent reading files, editing code, and running programs is added back, average total duration is 27.9 minutes — close to Copilot’s total. So the data support reduced time people must directly invest in the task; they do not directly translate into project delivery speed doubling.

Two further limits: time comparisons keep only correctly completed tasks, excluding failures and unfinished attempts; interaction logs are not attention monitoring and cannot tell whether users watched the screen while the Agent worked. Excluded runtime is not necessarily free time for other work.

On the two data-analysis problems in this experiment, only participants using the Agent finished. It can fetch data online and run processing scripts, reducing handoffs between tools. That also reminds us: results include differences in browser, terminal, and file access — they cannot all be attributed to the model writing code better.

03 Saved operations often sit after code is generated

Figure 4 groups operations into implementing features, configuring the environment, and debugging — blue for humans, orange for AI. Under both conditions, initial code was largely AI-generated; researchers even found Copilot use centered mainly on chat rather than continuously accepting autocomplete.

On environment setup, the paths diverge. Copilot users run commands themselves, paste errors into the chat assistant, then execute new suggestions. Some reported spending about eight minutes just getting the environment right. The Agent can try running, install dependencies, and retry on its own; researchers rarely saw users send messages specifically about environment setup.

Debugging is similar. What used to be people running, locating errors, and copying fixes can now be executed by the Agent, with people reviewing results and proposing corrections. The paper records interactions where users noted dark mode could not switch back, asked which folder to put changes in, and requested explanations for why a function was deleted.

The division-of-labor change is more concrete than “a few fewer lines typed.” Developers still must judge whether results meet requirements — they just need not hand-complete every step. The appendix’s manual analysis of 153 prompts to the Agent shows this work shift; it is not a test proving every operation is already reliably automated.

04 More tasks done right still did not buy consistent satisfaction

Figure 5 is the experience questionnaire — blue means agreeing that Agent is better, orange means disagreeing. Seventy-five percent of participants felt cognitive burden was lower; 70% felt it helped them complete new tasks they could not with prior tools. Authors found significant lean on both items, but these are self-reports, not direct measures of mental effort.

Speed, flow, and satisfaction were less tidy. Forty-five percent found Agent more satisfying; 40% disagreed; the satisfaction comparison did not reach significance. Another 55% felt they better understood Copilot’s output, but that item’s p = 0.07 also missed the paper’s 0.05 threshold.

Open responses were more specific. Some felt the Agent’s generation process was not visible enough and hard to control; with Copilot, reviewing suggestions and handling errors step by step actually helped them understand changes. That can show users hit understanding difficulties — it does not yet prove their code comprehension was harmed.

Figure 6 further asks whom they would use later. On prioritizing OpenHands over Copilot, 20% agreed and 60% disagreed. At the same time, 70% agreed OpenHands is more capable than human programmers. That last item is subjective evaluation only; the experiment did not pit the Agent against professional programmers working without AI.

Acknowledging it can finish hard problems, and being willing to hand daily work to it, are two different judgments. The questionnaire did not track long-term retention, but the split already appears. For tool designers, task success rates answer whether something got done — they may not explain why users want to keep using it.

05 This comparison saw early migration — not the full development cycle

Participants knew Copilot and were Agent novices. A familiar editor, existing habits, and an unfamiliar web interface can all shape preference. The authors acknowledge this. The 60% should not be written as developers universally rejecting Agents, nor predict how they would choose after months of fluent use.

Models were not fully equated either. OpenHands fixed Claude Sonnet 3.7; Copilot allowed model choice. The appendix found no significant task-performance differences among commonly used models, but splits leave smaller samples — one cannot assert model capability had no effect. Tool configuration and autonomous execution changed together here.

The experiment used only six prewritten problems, about 40 minutes per tool. It tested implementing tasks; it did not fully cover clarifying requirements or deciding what to build, nor track maintenance, code understanding, or rework days later. Equating passing tests with full software-development benefit exceeds the evidence.

The study also did not carefully record whether participants checked generated code line by line, what they changed, or effects of parallel work while waiting. Authors suggest developers may manage multiple Agents in the future — that is a later direction, not proof that this experiment already showed higher multitasking efficiency.

06 Agents need to let people take over — and know when to stop

The paper proposes three design demands: behavioral transparency, moderate initiative, and effective use of human input. They come from the process records and feedback above — not a validated new interface. Notably, feedback includes complaints that the Agent created large numbers of unexpected files, and requests to undo pushes made without clear consent.

When users ask why a function was deleted, they need explanations that map to actual files. I think a fluent completion summary is not enough: which changes have test support, and which are unverified, should be inspectable. Only then can developers decide to accept, keep investigating, or roll back. That is a design judgment extended from the paper’s feedback.

Installing necessary dependencies can save a round of back-and-forth; creating batches of extra files, committing, or pushing may cross what users originally expected. Actively finishing a task should not quietly become permission to expand its scope. Which actions can proceed autonomously, and which must pause to ask, should have clear boundaries in the interaction.

Then there are the moments of sitting and waiting. Participants already mentioned that experience. To turn the advantage of effort time down to 12.5 minutes into everyday benefit, tools still need to let people know whether they can leave now and when they must return to judge. This paper does not validate which reminder works best — it only puts the unsolved problem on the table.

Keeping a sense of control does not require reclaiming every command to run by hand. But after an Agent delivers, developers still need to know which files changed, which behaviors were tested, and whether unapproved operations were added. When those can be checked, time saved on execution more easily becomes work that can be handed off with confidence.

Paper information

Original title: Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows. Authors Valerie Chen, Ameet Talwalkar, Robert Brennan, and Graham Neubig, primarily from Carnegie Mellon University and All Hands AI. Published at CHI 2026; experiment conducted April–May 2025. Paper link: https://doi.org/10.1145/3772318.3790850

Key terms

Copilot workflow: people lead execution; AI provides suggestions, code, or conversational help. Agent workflow: AI can call tools and act continuously from runtime feedback. User effort time: in this study, participation time estimated from logs — not identical to total duration. Percentage points: the direct difference between two percentages; 25% to 60% is +35 percentage points.

References

Chen, V., Talwalkar, A., Brennan, R., & Neubig, G. (2026). Code with me or for me? How increasing AI automation transforms developer workflows. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI ’26) (pp. 1–19). Association for Computing Machinery. https://doi.org/10.1145/3772318.3790850