IUI 2026 | AI Screens 15 Papers Only 18% Faster — Why Researchers Still Prefer It

IUI 2026 | AI Screens 15 Papers Only 18% Faster — Why Researchers Still Prefer It cover

Fifteen papers took 10 minutes 58 seconds on average for manual screening; with AI-assisted Arc, the full workflow was about 9 minutes. Only 18% faster. On that number alone, it is hard to support a story that “AI remakes systematic reviews.”

But participants gave another, more interesting set of ratings: satisfaction with Arc was 6.25 versus 3.12 for the baseline tools; future-use intention rose from 3.00 to 6.12. What researchers care about is clearly not only fewer mouse clicks.

What I care about more is this gap: responsible AI does not have to do more for researchers. Repetitive labor can go to machines, but judgment grounds must stay inspectable at any time, and final decisions still belong to people.

01 What wears people down in systematic review is often not reading papers

The team first interviewed 20 computer researchers with systematic-review experience, spanning machine learning, HCI, databases, and related areas. They summarized three recurring problems: rewriting queries across databases is laborious; literature volume and update speed leave people constantly catching up; and the deeper automation goes, the more researchers worry about losing control of screening.

Figure 1 shows interviewees ranked reading/comprehension, coding/synthesis, and paper writing as most time-consuming. That is unsurprising. Later steps are easier to underestimate: keyword expansion, database choice, deduplication, and excluding irrelevant papers — none is the hardest alone, yet each forces researchers to switch tools, copy conditions, and check results. Attention is worn away little by little.

Figure 1: Twenty researchers’ ranking of how time-consuming systematic-review stages are. Reading/comprehension, coding/synthesis, and writing were judged most costly.

So the authors did not set the goal as “let AI finish the review.” They aimed at more concrete friction: how to reduce organizing and ferrying work without also automating researchers’ most important judgments.

02 Arc organizes the search process first, then brings in AI

Arc is a full web application that places Semantic Scholar, DBLP, Web of Science, IEEE Xplore, and Scopus search in one interface and supplements metadata with Crossref. Google Scholar was left out for a practical reason: no first-party API, unstable DOI returns, and difficulty guaranteeing precision, transparency, and reproducibility.

Its workflow includes keyword-variant management, search-version comparison, forward and backward citation chasing, and AI-assisted screening. Figure 2 strings these stages together. Arc’s design focus is not a bigger chat box, but leaving a trackable state for every retrieval, revision, and screening step.

Figure 2: Arc’s overall workflow, integrating cross-database search, search-version comparison, citation chasing, and verifiable AI screening in one interface.

Authors first handle deterministic frictions in the research process, then place generative AI in the screening stage that needs semantic judgment. The order cannot reverse. AI can suggest; it cannot quietly rewrite inclusion criteria for researchers.

03 Search diffs matter more than one “smarter” answer

Systematic-review search is not one submitted query; it is continual probing: swap a synonym, add a database, see what grew and what was missed. Traditional tools usually keep only the current result; the previous query version and the effect of changes easily scatter across sheets, browser tabs, and researchers’ memory.

Arc lets any two search versions be compared. In Figure 3, red marks removed conditions and papers, green marks additions, and retained results appear side by side. Researchers need not judge by impression whether a change was better; they can see how condition changes propagate into the literature set.

Figure 3: Arc compares conditions and results of two search versions side by side, highlighting added, removed, and retained papers.

Search diffs are less flashy than AI screening, yet I think they sit closer to this paper’s core contribution. They turn exploration from personal memory into inspectable research records. Systematic review values reproducibility; what must be kept is not only the final query string, but why researchers walked step by step to here.

04 AI’s job is to give a verifiable first judgment

In screening, researchers first define screening questions and labels, usually give about 3 examples, and write judgment reasons. Arc uses OpenAI o3-mini for few-shot inference, suggesting labels for later papers while pointing to abstract evidence and generating a rationale. Researchers can accept or override the judgment.

That “rationale” is not decoration. A correct classification can still rest on an unreliable explanation. The study saw exactly that: AI’s label was right, but the rationale did not align with the cues experts actually used. If the system shows only conclusions, such problems are hard to catch.

Arc moves researchers from making first judgments paper by paper to verifying first judgments paper by paper. Workload does not vanish; it changes place. Researchers save time hunting evidence from scratch, and spend it checking whether evidence is enough to support the label.

05 Being 18% faster is not the strongest result — cognitive load is

The paper then invited 8 computer researchers with systematic-review experience for a within-subjects comparison. Participants completed two fixed topic scenarios using Arc versus common baseline tools: Google Scholar, Google Sheets, and publisher pages. Tool order was randomized and balanced; scenario order was fixed.

Mean NASA-TLX on the search task fell from 4.04 baseline to 2.21 with Arc; snowball search from 4.75 to 1.96; screening from 3.67 to 2.75. Figure 4 shows the search-task dimension distributions; under Arc, mental demand, temporal pressure, effort, and frustration overall shifted toward lower scores.

Figure 4: In the search task, subjective cognitive load under Arc was overall lower than baseline.

Time changes were uneven. Snowball search shrank from a mean 10.68 minutes to under 1 minute, mainly because cross-page citation chasing and sheet organizing were taken over by the system. Screening 15 papers only fell from 10:58 to about 9 minutes, of which about 2 minutes waited on AI and researchers spent about 7 minutes verifying.

That unspectacular 18% at the opening only makes sense here. As long as people carefully review every suggestion, AI cannot squeeze screening into a few seconds. What it mainly changes is attention allocation: less ferrying, more judging.

06 95% accuracy cannot be read as AI already doing reviews

In this controlled screening of 15 papers, independent human classification accuracy was 75%; AI-suggested label accuracy was 95%. The result is striking, but it describes only a small-batch, short-duration task, and every participant verified AI output item by item. It cannot imply the model can independently finish a systematic review, still less equate to higher long-term review quality.

Researchers’ sense of assurance supplies another line of evidence. Early in the task, conditions were close: baseline 6.25, Arc 6.50. By the final stage, baseline fell to 4.75 while Arc held at 7.00. Figure 5 shows the gap between tools widening as steps advanced.

Figure 5: As the task progressed, assurance under baseline declined while Arc stayed high.

Still, this is an 8-person, roughly one-hour experiment; all participants came from computing. The study did not cover multi-month real reviews, did not test multi-person collaboration, continually changing criteria, reviewer agreement, or automation bias, and did not run full literature-recall evaluation. External database API coverage likewise limits the search boundary.

One more striking reality: the team spent 6 months getting an IEEE Xplore API key, and ACM Digital Library lacks a robust public bulk interface. Even the best interaction design cannot escape academic infrastructure permissions and availability.

07 Responsible design lands in five concrete choices

Arc’s exploratory results are not enough to prove a universal solution, but they offer five transferable design choices: keep search history; show version diffs; have AI give labels and evidence together; make human verification part of the workflow; allow import/export in open formats such as CSV, BibTeX, and RIS.

Machines handle cross-database formats, result alignment, and initial classification; people define criteria, check evidence, and handle exceptions. Evaluating such systems cannot only ask “how much faster.” One must also ask whether researchers can understand where a suggestion came from, catch where it is wrong, and export and recheck the whole process.

What is truly worth pursuing is not an unattended review, but letting researchers spend time again on research judgment. This paper does not prove the destination has been reached, but it says more clearly than a larger automation promise how the next step should be designed.

Paper information

Original title: From Toil to Thought: Designing for Strategic Exploration and Responsible AI in Systematic Literature Reviews

Authors: Runlong Ye, Naaz Sibia, Angela Zavaleta Bernuy, Tingting Zhu, Carolina Nobre, Viktoria Pammer-Schindler, Michael Liut

Main affiliations: University of Toronto, McMaster University, University of Toronto Mississauga, Graz University of Technology

Paper link: https://doi.org/10.1145/3742413.3789079

Key terms

Systematic review: Retrieving, screening, and synthesizing literature under predefined methods, with emphasis on process transparency and auditability — not merely reading many papers.

Snowball search: Continuing to find related work along a paper’s references and later papers that cite it.

NASA-TLX: A common scale for subjective task load, including mental demand, temporal pressure, effort, frustration, and related dimensions.

References

Ye, R., Sibia, N., Zavaleta Bernuy, A., Zhu, T., Nobre, C., Pammer-Schindler, V., & Liut, M. (2026). From Toil to Thought: Designing for Strategic Exploration and Responsible AI in Systematic Literature Reviews. In Proceedings of the 31st International Conference on Intelligent User Interfaces (IUI ’26). ACM. https://doi.org/10.1145/3742413.3789079