Short answer: A chatbot can run short, text-based qualitative interviews cheaply and at scale, but not yet deep ones. In a 2026 study, a Gemini-powered chatbot interviewed 74 social scientists for a total model cost of €0.04 and produced 149 pages of data, yet the authors concluded it did not reach the depth and nuance a qualitative interview usually aims for.
AI chatbot interviews promise something qualitative researchers have rarely had: dozens of adaptive, semi-structured conversations running at the same time, with no scheduling and almost no marginal cost. A new open-access article by Tullia Jack, Alex Cooper and Lisa Flower in Methodological Innovations (2026) tests that promise on an unusually demanding audience: professional social scientists who know what a good interview looks like. Their verdict is useful precisely because it is mixed.
What did the study test?
The study asked whether an LLM can work as a qualitative interviewer, using a chatbot to interview researchers about their own use of generative AI. The paper is Automating the qualitative interview? Using Gen AI chatbots in social science research (Methodological Innovations, Vol. 19, Issue 3, first published May 27, 2026).
The authors built their own web chatbot on Google's Gemini 1.5 Flash, accessed through the API, and prompted it to follow a semi-structured topic guide with four themes: current use of generative AI, existing coding practices, ideal future uses of AI in analysis, and ethics. The prompt told the model to open each theme with a broad question, probe with follow-ups, avoid sensitive topics, and move on if someone declined to answer. Participants were told the chat would take around 10 minutes.
How was the data collected?
- Pilot ("Percival"): 23 attendees of a workshop on qualitative analysis with generative AI. Their feedback made the bot less relentlessly positive and added a final catch-all question.
- Main round ("Elsa"): 51 researchers from a social science faculty at a Swedish university, recruited through heads of department.
- Follow-up: 39 of the 51 agreed to be contacted, and 23 replied by email with reflections on being interviewed by a bot.
That gives 74 chatbot interviews plus 23 follow-up emails. Humans, not the model, did the main analysis.
How well did the chatbot interview?
On efficiency, it performed very well. According to Jack, Cooper and Flower (2026), the whole project cost €0.04 in model fees (excluding researcher salaries), most conversations lasted 8 to 12 minutes, and more than four-fifths of participants finished the interview. Completed chats ranged from 1 minute 49 seconds to 33 minutes 20 seconds.
The bot also showed real interviewing skill. It summarised points back for confirmation, suggested ways forward when someone seemed stuck, recognised field-specific acronyms, and followed participants into topics outside the guide. The authors saw no hallucinations in the chats, and it stayed away from sensitive topics as instructed. In a footnote, the first author writes that she was humbled by how well the bot got people talking.
Where did the AI interviewer fall short?
The failures were mostly about conversational pacing and leading. The authors document several patterns that a skilled human interviewer would avoid:
- Interrupting: the bot replied the moment someone pressed enter, so people who think as they type lost their train of thought.
- Stacked questions: it sometimes asked several questions at once and then dropped the ones the participant did not answer.
- Missed probes: it passed over many chances to follow up, and it was not always clear why it probed or moved on.
- Rigid guide-following: it kept asking quantitative researchers about qualitative methods, which irritated at least one participant.
- Leading premises: it seemed to assume qualitative analysis is time-consuming, which nudged several participants toward discussing time savings.
- Tone: some found it repetitive or too effusive in thanking them after every answer.
Of the 23 follow-up emails, 11 were broadly positive, 8 were neutral or mixed, and 4 described feeling disengaged or mildly annoyed. Some participants said talking to a bot made it easier to be blunt and honest.
AI chatbot interviews vs human interviews: what the evidence shows
The study's findings line up as a trade-off between scale and depth.
| Dimension | What the chatbot achieved | What it lost |
|---|---|---|
| Cost and logistics | €0.04 in model fees for 74 interviews; simultaneous, no scheduling | Needed custom web development, API work and prompt iteration |
| Volume | 149 pages of text, a little under half of it from participants | The authors stress that quantity is not the measure of a good interview |
| Probing | Summarised, clarified and handled expert vocabulary | Missed probes, stacked questions, cut people off |
| Neutrality | Perceived as non-judgmental; some answered more candidly | Introduced at least one leading assumption |
| Attention | Participants could answer whenever suited them | Signs of multitasking, such as one sporadic 51-minute chat |
The authors' bottom line is direct: chatbots may suit some research questions, perhaps as a complement to questionnaires, but are currently not appropriate for gathering thick data.
Can the same LLM analyse the interviews?
Only as a starting point. The authors gave all 74 transcripts to Gemini and asked for Weberian ideal types of how social scientists relate to AI. The model proposed five types; the researchers merged overlapping ones into three (Enthusiasts, Pragmatic Adopters and Gen AI Agnostics). The quotes the model cited were accurate about 80% of the time, and it claimed younger researchers were more enthusiastic, a pattern the authors could not find in the data. They also note how tempting it was to keep re-prompting rather than stop and reflect.
The participants were in a similar place. Of the 74, 37 were experimenting with generative AI, mostly for writing support, translation, transcription and literature searches. Few used it for analysis, and many worried about black-box coding, reproducibility and data security.
What this means for researchers using AI interviewers
The study works as a checklist of failure modes to test for before fielding an AI-moderated study:
- Let people finish. Text interviews need a way to send several messages before the AI replies, or a spoken format, which the authors suggest exploring.
- One question per turn. Write this into the prompt and check pilot transcripts for it.
- Audit the guide for hidden premises. If the AI "believes" something about the topic, it will lead with it.
- Add skip logic for misfit participants. Don't let the guide push a quantitative researcher through qualitative questions.
- Calibrate tone. Constant praise reads as insincere to many participants.
- Be honest about length. One participant pointed out that "a few minutes" had become over a quarter of an hour.
- Verify every quote in AI analysis. At roughly 80% quote accuracy, every excerpt needs checking against the source.
Most of these can be caught in a pilot. Our guide to pilot-testing an AI interview guide covers how to run one. If you use a hosted tool such as the Qualitati AI Interviewer rather than building your own bot, the same tests apply: read pilot transcripts specifically for interruptions, stacked questions and leading framing before you scale up.
Two limits matter when reading the results. The sample was small and self-selected, and the authors expect many participants were already interested in AI. The bot also ran on Gemini 1.5 Flash, and newer models may behave differently.
FAQ
How much does an AI chatbot interview cost?
Model costs can be tiny. In Jack, Cooper and Flower (2026), 74 text interviews cost €0.04 in API fees. That excludes development, prompt design and researcher time, which were the real costs.
Can AI chatbots replace human qualitative interviewers?
Not for in-depth work, according to this study. The chatbot gathered a lot of data and engaged participants, but the authors judged that it did not reach the depth and nuance of a good qualitative interview.
Do people answer more honestly to a chatbot?
Some participants said so. Several described the bot as non-judgmental and easier to be blunt with, though others felt disengaged or found it odd that an AI was collecting the data.
Which model did the study use?
Google's Gemini 1.5 Flash, accessed through the API inside a custom Python and JavaScript web application.
Last updated: October 6, 2026. This is an independent editorial summary of third-party research published in Methodological Innovations; Qualitati is not affiliated with the authors. See the original article for full methods and findings.