Can large language models produce reliable thematic analysis? A December 2025 study says yes — when you run the same model several times and measure agreement. Testing Gemini 2.5 Pro, GPT-4o, and Claude 3.5 Sonnet on an interview transcript, all three exceeded the κ > 0.80 threshold for “excellent” inter-rater reliability, with Gemini scoring highest.
\n\nThe paper, “Multi-LLM Thematic Analysis with Dual Reliability Metrics” by Nilesh Jain, Hyungil Suh, Seyi Adeyinka, Leor Roseman, and Aza Allsop (arXiv, December 2025; revised February 2026), asks a practical question: if you ask an LLM to code a qualitative transcript, how do you know the output is trustworthy rather than a one-off hallucination?
\nTheir answer is an ensemble approach. Instead of running a model once, they ran each model six independent times on the same psychedelic art-therapy interview transcript, varying random seeds (1–6) and temperature (0.0–2.0). They then measured how consistently each model reproduced the same themes across runs, using two complementary metrics.
\n\nThe study used two metrics that capture different kinds of agreement:
\nCombining the two is the paper’s core methodological contribution: kappa alone penalizes harmless rewording, while semantic similarity alone can mask genuine coding disagreement. Together they give a fuller picture of reliability.
\n\nAccording to Jain et al. (2025), all three models cleared the κ > 0.80 bar, but Gemini led on both metrics:
\n| Model | Cohen’s Kappa | Cosine Similarity | Consensus Themes |
|---|---|---|---|
| Gemini 2.5 Pro | 0.907 | 95.3% | 6 (50–83% consistency) |
| GPT-4o | 0.853 | 92.6% | 5 |
| Claude 3.5 Sonnet | 0.842 | 92.1% | 4 |
The gap between models is real but modest. More striking is that every model surpassed the excellence threshold once results were aggregated across runs — a reminder that the reliability gain comes as much from the ensemble method as from the choice of model.
\n\nBecause LLMs are probabilistic, a single run can surface an idiosyncratic reading of a transcript. That is the same worry a qualitative researcher has about a single human coder. The classic fix in qualitative methods is a second coder and an inter-rater agreement statistic. The study reframes that fix for AI: treat each model run as a separate “rater,” then require themes to survive across runs before you trust them. Gemini’s six consensus themes, for instance, each recurred in 50–83% of runs — the themes that appear once and vanish are exactly the ones you should be skeptical of.
\n\nThe takeaway is not “use Gemini.” It is that AI-assisted coding needs a reliability protocol, not a single prompt. Three practices transfer directly to everyday work:
\nThis is the workflow behind AI-assisted qualitative tools generally. If you want to try structured, auditable theme extraction on your own transcripts, ThemeLens applies AI thematic analysis with a transparent coding trail, and QualiTaTi’s AI Interviewer can generate the transcripts to analyze in the first place.
\n\nThis was a single interview transcript in one domain, so the exact kappa values should not be read as universal model rankings. The metrics validate internal consistency across runs — they do not verify that the themes match a human expert’s interpretation or the participant’s intended meaning. Reliability and validity remain distinct questions, and the paper measures the former.
\n\nIs AI thematic analysis reliable?
In this study, yes — when aggregated over multiple runs, all three tested LLMs reached Cohen’s κ above 0.80, the conventional threshold for excellent agreement. Reliability from a single run is not guaranteed.
Which LLM did best?
Gemini 2.5 Pro (κ = 0.907, 95.3% semantic similarity), narrowly ahead of GPT-4o (0.853) and Claude 3.5 Sonnet (0.842).
Does high reliability mean the coding is correct?
No. The metrics measure consistency across runs, not agreement with human interpretation. A human researcher should still validate that the themes are meaningful and accurate.
How many runs should I use?
The study used six per model with varied seeds and temperature. The principle — run multiple times and keep only themes that recur — matters more than the exact count.
Last updated: July 27, 2026. This is an independent editorial summary of third-party research; QualiTaTi is not affiliated with the study’s authors. Read the original paper: Multi-LLM Thematic Analysis with Dual Reliability Metrics (arXiv:2512.20352).
","related":[{"slug":"ai-conversational-interviewing-2026-study","title":"Can AI Conduct Research Interviews? A 2026 Field Study","description":"Wuttke et al. (2026) ran AI-led interviews with 571 respondents on migration attitudes: richer reasoning than surveys, and ratings at or above the survey. Delivery mode matters.","category":"Guides","date":"2026-07-31","readTime":"8 min read","author":"Qualitati Research Team"},{"slug":"digital-twins-survey-respondents-2026-study","title":"Can Digital Twins Replace Survey Respondents? A 2026 Study","description":"A 2026 Columbia Business School study found LLM digital twins correlate with real survey answers at only r = 0.20 across 164 outcomes: funhouse mirrors, not substitutes.","category":"Guides","date":"2026-07-29","readTime":"8 min read","author":"Qualitati Research Team"},{"slug":"is-ai-qualitative-coding-biased-2025","title":"Is AI Qualitative Coding Biased? What a 2025 Study Found","description":"A 2025 Sociological Methods & Research study found ChatGPT and Llama 2 coding errors correlated with respondent traits; a small supervised model was more accurate and less biased.","category":"Guides","date":"2026-07-28","readTime":"8 min read","author":"Qualitati Research Team"}]}