Do AI survey probes improve data quality? A 2025 randomized experiment with 1,800 respondents found that AI chatbot probing produced longer, more detailed open-ended answers and coded them into pre-set categories better than chance without any fine-tuning — but at a slight cost to respondent experience and with false positives inflated by acquiescence bias. AI probing helps, with guardrails.
\n\nThe study built a framework for \"AI-assisted conversational interviewing\" and tested it in a live web survey. According to Barari et al. (2025), 1,800 participants recruited from a non-probability online panel were randomly assigned to AI chatbots powered by large language models. The chatbots did two jobs at once: they dynamically probed respondents for elaboration on open-ended questions, and they coded those answers into researcher-defined categories in real time — a task usually done by humans after fielding.
\nRespondents were split across a control condition and two probing treatments: Treatment 1 asked confirmation-style probes (\"Just to confirm, did you mean…?\"), and Treatment 2 asked elaboration or relevance probes that pushed for more detail. This design lets us separate the effect of asking more from the effect of the AI simply being present.
\n\nReasonably accurate — better than random, but not human-grade on every question. According to Barari et al. (2025), the chatbots \"performed better than random guesses\" on live coding across most questions without survey-specific fine-tuning. The clear exception was news-source questions, where both accuracy and recall fell below 70%. For context, the authors note prior LLM evaluations have reported 91–94% classification accuracy on open-ended survey responses, so live, un-tuned coding sits below that ceiling.
\nThe catch is how accuracy was measured. When respondents themselves confirmed the AI's coding, agreement looked high. But when trained human coders re-checked the same responses, precision was on average 20% lower than the respondents' own confirmations. That gap is the fingerprint of acquiescence bias: people tend to say \"yes\" to a confirmation probe even when the category is slightly off.
\n\nBoth, with a trade-off. Open-ended responses under probing were \"more detailed and informative,\" per Barari et al. (2025). But richer data came at a cost to the respondent's experience, and probing raised friction: answer changes and back-tracking to earlier items were roughly twice as common in the confirmation-probing condition as in the control (88 vs. 43 instances), and interview duration was longer for both treatments — the elaboration-probing chats lasted about twice as long on average.
\n\n| Dimension | What the study found | Implication for researchers |
|---|---|---|
| Answer richness | More detailed, informative open-ended responses | Better raw material for thematic analysis |
| Live coding accuracy | Above chance without fine-tuning; <70% on news sources | Treat AI codes as a draft, not a verdict |
| False positives | Inflated by acquiescence on confirmation probes | Avoid leading \"did you mean…?\" phrasings |
| Human vs. self-agreement | Coder precision ~20% below respondent confirmation | Validate a sample against human coders |
| Respondent experience | Slightly worse; longer interviews, more back-tracking | Cap probe depth; probe only where it pays |
The headline for practitioners: AI probing is a genuine data-quality lever, not a gimmick — but it is not free. Three takeaways follow directly from the evidence.
\nThese lessons map onto how modern platforms are built. Qualitati's conversational surveys use AI-driven follow-up questions and branching so open-ended answers get probed in the moment, and its work on AI follow-up questions reflects the same finding — good probing deepens data, poor probing just adds burden. For turning those richer transcripts into themes, ThemeLens keeps a researcher in the analysis loop rather than treating a first-pass AI code as final.
\n\nThis study sits alongside a growing body of work on where AI helps and hurts in qualitative and survey research. It echoes findings that AI can code open-ended data competently but drifts on ambiguous categories, and that agreement metrics can flatter AI if you measure them the wrong way. For a related look at whether AI models agree with each other on themes, see our summary of multi-LLM thematic analysis.
\n\nNot entirely. In this study the AI coded better than chance without fine-tuning, but human coders disagreed with roughly a fifth of the confirmations respondents accepted. AI is a strong first pass that still needs human validation on a sample.
\n\nSlightly. Probing lengthened interviews and increased back-tracking and answer changes, and overall experience dipped a little. The effect was manageable, but it argues for probing only where the extra detail is worth the burden.
\n\nMostly acquiescence bias. When the chatbot asks \"did you mean X?\", respondents lean toward \"yes\" even when X is not quite right, so categories get confirmed that a neutral human coder would reject.
\n\n1,800 participants, randomly assigned across a control condition and two AI-probing conditions in a web survey experiment, per Barari et al. (2025).
\n\nLast updated: July 11, 2026.
\nThis article is an independent editorial summary of third-party research (Barari, S., Angbazo, J., Wang, N., Christian, L. M., Dean, E., Slowinski, Z., & Sepulvado, B. (2025). \"AI-Assisted Conversational Interviewing: Effects on Data Quality and Respondent Experience,\" arXiv:2504.13908). Qualitati is not affiliated with the authors, and all figures are drawn from the cited preprint.
\n","related":[{"slug":"ai-vs-human-interviewers-randomized-chat-experiment-2026","title":"AI vs Human Interviewers: What a 2026 Randomized Test Found","description":"A 2026 randomized study compared AI vs human interviewers in text chat: equal relevance, no detectable specificity gap, 65% more text, but more skipped questions.","category":"AI User Research","date":"2026-10-19","readTime":"8 min read","author":"Qualitati Research Team"},{"slug":"artificial-societies-benchmark-synthetic-population-validity-2026","title":"Is Your Synthetic Population Valid? An 11-Test Benchmark (2026)","description":"Is your synthetic population valid? A 2026 benchmark tested 9 LLMs on 11 validity checks: they match averages but compress scales and distort trait links.","category":"AI User Research","date":"2026-10-15","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"synthetic-consumers-visual-marketing-pretest-2026","title":"Can Synthetic Consumers Pretest Logos and Ads? A 2026 Study","description":"Can synthetic consumers pretest visual marketing? A 2026 study found GPT agents saw logos correctly but matched at most 2 of 6 human effects and reversed one.","category":"AI User Research","date":"2026-10-14","readTime":"7 min read","author":"Qualitati Research Team"}]}