Last updated: June 28, 2026
\n\nPutting a human \"in the loop\" does not automatically fix AI annotation. In a pre-registered 2025 experiment with 410 annotators and over 7,000 annotations, Schroeder, Roy, and Kabbara (MIT) found that showing annotators an LLM's suggested label anchored their judgment, shifted the overall label distribution, and made people more confident — without making them faster. Critically, when those AI-assisted labels were then used to score the same AI, the model's measured performance was inflated. Human review is necessary, but it is not a guarantee of validity.
\n\nThe researchers ran a pre-registered online experiment on subjective text-annotation tasks — the kind where reasonable coders disagree. They recruited 410 crowdworker annotators who produced more than 7,000 annotations across conditions that varied whether, and how, an LLM suggestion was shown alongside each item. Because the tasks were subjective, there was no single ground-truth label, so the team measured how the distribution of human labels moved when an AI suggestion was present versus a baseline with no suggestion.
\n\nA lot. According to Schroeder, Roy, and Kabbara (2025), annotators \"strongly took the LLM suggestions, significantly changing the label distribution compared to the baseline.\" In other words, the same person shown an AI-proposed label tends to ratify it rather than independently re-judge the item. This is consistent with the broader literature on automation bias and anchoring: when a confident-looking default is on screen, people drift toward it.
\nTwo secondary effects matter for research operations. First, annotators reported higher self-confidence when given LLM suggestions — they felt more sure even as their independence eroded. Second, the suggestions did not speed annotation up, so the common justification (\"a human glance is fast and cheap\") loses one of its main selling points.
\n\nHere is the circularity the paper highlights. Suppose you generate labels with a human-plus-LLM workflow, then use those same labels as the benchmark to measure how good the LLM is. Because the human labels have already been pulled toward the LLM's outputs, the model is effectively being graded against an answer key it helped write. The study found that \"reported model performance significantly increases\" under this setup. Any downstream analysis built on those labels — an effect size, a prevalence estimate, a theme frequency — inherits the same distortion.
\n\n| Dimension | AI-suggests-first (review mode) | Human-first (blind, then reconcile) |
|---|---|---|
| Anchoring risk | High — label distribution shifts toward the model | Low — human judgment recorded before seeing AI |
| Speed | No measured gain (Schroeder et al., 2025) | Slower up front, but auditable |
| Confidence calibration | Inflated self-confidence | Truer to actual disagreement |
| Safe to evaluate the AI? | No — circular, inflates metrics | Yes — independent reference labels |
| Best use | High-volume, low-stakes triage | Reliability checks, validity-critical coding |
Qualitative coding is a subjective annotation task by another name. If your codebook involves interpretation — sentiment, motivation, stance, or theme assignment — the same anchoring dynamics apply when an AI proposes codes and a human \"approves\" them. Three practical implications:
\nThis is why tools that support qualitative analysis should make the workflow explicit. In Qualitati's QDA Workspace and ThemeLens thematic-analysis pipeline, AI-assisted coding is designed to keep an audit trail and to let researchers reconcile AI suggestions against their own judgment rather than silently inheriting them — so that reliability can be checked, not assumed.
\n\nThe study used crowdworkers on short subjective tasks, not trained qualitative researchers working a deep codebook, so the size of the anchoring effect among experts is an open question — experts may resist more, or may anchor just as readily under time pressure. The work is a 2025 arXiv preprint; readers should confirm its status when citing. And \"blind-first\" annotation costs time and money, so the right balance between speed and independence depends on how validity-critical the labels are.
\n\nNot in this study. According to Schroeder et al. (2025), LLM suggestions did not make annotators faster, even though annotators felt more confident.
\n\nIt is the tendency for human annotators to shift their labels toward an AI's suggested label simply because it is shown. The 2025 study found this significantly changed the overall label distribution versus a no-suggestion baseline.
\n\nYou should not. Because those labels are pulled toward the model's own outputs, they inflate the model's measured performance. Use independent, AI-naive reference labels for evaluation.
\n\nCollect some blind human codes before showing the AI's suggestions, keep an audit trail, and separate the people who generate labels from the reference set used to measure reliability.
\n\n\"Just put a human in the loop\" is not a validity strategy on its own. The 2025 evidence is that AI suggestions anchor human judgment, raise false confidence, and — if you are not careful — quietly inflate the very metrics you use to trust the AI. For qualitative work, the fix is procedural: judge blind first, reconcile second, and never grade an AI against labels it helped produce.
\n\nPrimary source: Schroeder, H., Roy, D., & Kabbara, J. (2025). Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks. arXiv:2507.15821.
\n\nThis article is an independent editorial summary of third-party research. Figures and findings are attributed to the cited authors; Qualitati is not affiliated with them. Last updated: June 28, 2026.
\n","related":[{"slug":"synthetic-survey-panels-marginal-fidelity-2026","title":"Do Synthetic Survey Panels Simulate People? A 2026 Test","description":"Do synthetic survey panels simulate real respondents? A 2026 test finds matching published averages is not proof: a no-persona prompt beat panels 9 of 9 times.","category":"Methodology","date":"2026-10-22","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"llm-as-judge-qualitative-interpretation-2026","title":"Can LLM-as-Judge Rate Qualitative Interpretations? A 2026 Test","description":"Can LLM-as-judge replace human review of qualitative interpretations? A 2026 study of 3,560 AI interview readings finds it screens models but misses nuance.","category":"Methodology","date":"2026-10-21","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"rehearse-interview-protocol-simulated-participants-2026","title":"Can You Rehearse Interviews with AI Participants? A 2026 Study","description":"Can you rehearse an interview guide with AI-simulated participants? A 2026 study of 21 researchers found real protocol revisions and three hidden risks.","category":"Methodology","date":"2026-10-21","readTime":"7 min read","author":"Qualitati Research Team"}]}