Short answer: Yes, with caveats. In a 2026 study of 903 open-ended answers from a European PhD student survey, GPT-5.4 reached an Adjusted Rand Index of 0.61 with human coders at the code level and 0.54 at the theme level. That is moderate agreement, close to how consistent each coder was with itself, but it swung from 0.31 to 0.85 depending on the question.
Most evidence that LLMs can code qualitative data comes from deductive tasks, where the model sorts text into categories a researcher already defined. Inductive content analysis is harder: the codes and themes have to emerge from the data. A new preprint by Bergmann, Gheorghiu, Gvritishvili, Mican, Stewart and Tolonen-Weckström (2026) puts that harder task to a direct test on real survey text, and the answer is more useful than a simple yes or no.
What did the study test?
The study compared human and LLM inductive coding of the same open-ended survey answers. The paper is LLMs for Survey Text Analysis: A Performance Comparison Between Humans and GPT-5 on Inductive Content Analysis (arXiv, August 2026), led by Leonardo Bergmann at the University of Vienna with co-authors from universities in Romania, Georgia, Czechia, Ireland and Sweden.
The data came from a European Students' Union survey of 2,800 PhD students. The authors drew a random 10% of responses to six open-text questions, giving 903 answers on topics such as how to improve doctoral students' financial situation, what would most improve well-being, and what mobility barriers they face. The question sizes ranged from 266 answers down to 55.
How was the comparison set up?
- Human coding. Five human coders each took one question (one coder handled the last two). They followed a written instruction: skim all answers, assign one or more short codes per answer, reuse existing codes where possible, then group codes into themes at a "not too specific, not too broad" level.
- LLM coding. GPT-5.4 was called through the OpenAI API with a four-prompt "two-times two-step" procedure: generate codes from the answers, assign each answer to one or more codes, generate overarching themes from the codes, then assign each code to a theme. The full prompts are published in the paper's appendix.
- Agreement metric. Because answers could carry several codes, the authors turned each coder's output into a co-assignment matrix and compared partitions with the Adjusted Rand Index (ARI). ARI is 0 for random grouping and 1 for perfect agreement, so it works even when human and AI codes use different labels.
- Qualitative check. Each human coder reviewed the LLM's output and said whether they would trust it for the rest of the dataset.
How closely did GPT-5.4 match human coders?
On average, GPT-5.4 and the humans agreed moderately. According to Bergmann et al. (2026), the mean ARI between human and LLM was 0.61 for codes and 0.54 for themes. For context, the alignment between each entity's own code-level and theme-level groupings was 0.68 for humans and 0.76 for the LLM. The authors read this as evidence that the human–LLM gap is not much larger than the variation inside a single coding entity.
The averages hide large differences between questions, which is the most practical finding in the paper:
| Survey question | Answers | Human–LLM (codes) | Human–LLM (themes) |
|---|---|---|---|
| Accessibility increase | 266 | 0.44 | n/a |
| Financial situation | 199 | 0.35 | 0.31 |
| Well-being | 173 | 0.72 | 0.41 |
| Mobility barriers | 146 | 0.79 | 0.65 |
| Personal experience | 64 | 0.51 | 0.64 |
| Anything else | 55 | 0.85 | 0.70 |
Source: Bergmann et al. (2026), Table 1. ARI values; the theme value for accessibility is missing because that coder did not produce themes.
Why does agreement vary so much between questions?
Agreement dropped wherever either coder was internally inconsistent. The authors report that low within-entity consistency always went together with low human–LLM agreement. On the financial-situation question, for instance, the LLM's own code-to-theme consistency was just 0.53, and human–LLM theme agreement fell to 0.31. On personal experience, the human coder's own consistency was the weak point (0.46).
The human coders' comments explain part of this. The financial-situation coder noted that grants, scholarships and salaries point in the same direction but differ in nuance, which makes the categories genuinely ambiguous. That coder also observed that the model tended to assign substantive codes to answers the human would have marked "unclear", in effect creating signal from noise. The well-being coder flagged three issues: over-interpreting short, vague answers; collapsing distinctions the human kept separate; and broadening themes until they lost some usefulness.
Codes or themes: where is AI more reliable?
AI was more reliable at the code level than at the theme level. Codes capture the basic meaning of a single answer; themes require an extra round of abstraction with more degrees of freedom. The authors note that humans also agree less on themes than on codes, so the drop is not unique to the model. In practice, it means the theme layer is where researcher review time is best spent.
What this means for researchers
The study supports using an LLM for the first, labor-intensive pass of inductive coding on open-ended survey data, with humans owning interpretation. All five human coders said the LLM output reflected their own categorizations and that they would support using the approach on the remaining data. The authors are explicit that this does not mean LLMs replace qualitative judgment.
- Check consistency per question, not just overall. A single average can hide a question where agreement is 0.31. Run agreement checks separately for each open-ended item.
- Watch ambiguous and thin answers. Questions with overlapping concepts (like funding types) and short, vague replies produced the weakest alignment. Keep an "unclear" option and check whether the model is over-coding.
- Spend review time on themes. Accept or lightly edit code-level output, then scrutinize how codes are grouped.
- Pick a label-free agreement metric. Inductive coders invent different labels for the same idea; ARI on co-assignment matrices sidesteps that, where Cohen's kappa cannot.
If you analyze open-ended survey responses at scale, an AI-assisted workflow like ThemeLens lets you generate codes and themes inductively and then review and merge them yourself. You can also collect richer open-text data in the first place with AI Surveys, which add conversational follow-ups to open questions.
What are the study's limitations?
The evidence is promising but narrow. The authors list the main limits themselves: one model (GPT-5.4), one prompting procedure, no ground truth, and a single human coder per question, so human–human agreement on the same answers was not measured. The "within" consistency figures compare a coder's code layer with its own theme layer, not two independent coders. The sample is one survey of PhD students, and results may differ for interview transcripts or other populations. All codings and the R analysis code are openly available on OSF, which makes replication straightforward.
FAQ
Can ChatGPT do inductive content analysis?
In this 2026 study, GPT-5.4 produced inductive codes that moderately matched human coders (ARI 0.61) on 903 survey answers. It worked best as a first-pass tool, with humans reviewing themes and ambiguous answers.
What is the Adjusted Rand Index in qualitative coding?
The Adjusted Rand Index measures whether two coders group the same pairs of responses together, regardless of the labels they use. It is 0 for chance-level grouping and 1 for identical grouping, which makes it suitable for comparing inductive codes.
Is AI better at codes or themes?
Codes. Agreement with humans was 0.61 for codes and 0.54 for themes, because theme building adds a further layer of interpretation where both humans and models diverge more.
Should I still have humans code open-ended survey responses?
Yes. The authors recommend LLMs for the early stages of inductive coding while human researchers keep responsibility for interpretation, validation and theory-building.
Last updated: 2026-10-05.
This article is an independent editorial summary of third-party research. Qualitati is not affiliated with the study's authors.