Can large language models reliably code open-ended survey responses? Not out of the box. A 2025 study in Survey Research Methods tested GPT, Llama, and Mistral on 5,072 German survey answers and found that only a fine-tuned model reached professional coding quality (macro F1 of 0.87). Zero- and few-shot prompting fell well short of trained human coders and even a classic machine-learning baseline.
\n\nResearchers Leah von der Heyde, Anna-Carolina Haensch, Bernd Weiß, and Jessica Daikeler asked a focused question: can off-the-shelf LLMs classify open-ended survey responses into a predefined coding scheme as well as human experts? They used real data from the GESIS Panel — a random 20% sample of 5,072 responses to a question about why people participate in surveys. Two survey researchers had already coded this data independently with very high agreement (Cohen's kappa = 0.91), giving a trustworthy human benchmark across 22 categories.
\nThe team compared three multilingual models — GPT-4o (proprietary), Llama 3.2, and Mistral NeMo (both open-source) — across several prompting strategies: zero-shot, few-shot, category descriptions, and fine-tuning. Because the categories were unevenly distributed, they used macro F1 (which weights rare categories equally) as the primary metric rather than raw accuracy.
\n\nAccuracy depended heavily on both the model and the prompting approach. According to von der Heyde et al. (2025):
\nThe models were not uniformly good or bad — they were lopsided. GPT coded common, concrete categories very well (macro F1 around 0.9 for incentive, interest, and fun) but collapsed on ambiguous or rare ones (macro F1 between 0.02 and 0.3 for no reason, non-identifiable/other, and other survey characteristics). Because performance varied so much by category, the AI-generated distribution of reasons diverged from the human-coded distribution — meaning that reported frequencies, not just individual labels, can be distorted when fine-tuning is skipped.
\n\n| Setup | Macro F1 (GPT) | Verdict |
|---|---|---|
| Zero-shot prompting | ~0.70 (worst of the three) | Below supervised baseline |
| Few-shot prompting | 0.71 | Best without fine-tuning, still short of SVM |
| Fine-tuning | 0.87 | Only \"satisfactory\" setup |
| SVM baseline (2022) | 0.83 (median F1) | Beat all non-fine-tuned LLMs |
The headline pattern: model choice mattered more than prompt engineering. Few-shot beat zero-shot across the board, but the size of that gain depended on the model, and no amount of prompting closed the gap between open-source models and GPT. The single biggest lever was fine-tuning — retraining GPT on labeled examples — which is also the most resource-intensive to do.
\n\nThe study is a useful corrective to the assumption that a capable general model plus a good prompt equals reliable coding. Three practical takeaways:
\nIf you code open-ended survey data or interview transcripts, treating the model as an assistant that proposes codes for human confirmation — rather than an autonomous coder — matches the evidence. Tools like QualiTaTi's ThemeLens are built around that human-in-the-loop pattern for AI thematic analysis, and its AI Surveys feature keeps researchers in control of how open-ended answers are interpreted.
\n\nCan ChatGPT code open-ended survey responses accurately?
Only partially. In this 2025 study, GPT-4o with few-shot prompting reached macro F1 of 0.71 — decent but below a trained SVM and short of expert-level reliability. Fine-tuning was needed to reach 0.87.
Are open-source LLMs good enough for survey coding?
Not in this test. Llama 3.2 and Mistral NeMo trailed GPT substantially and failed entirely on some categories, even with their best prompting setup.
Does prompt engineering fix weak AI coding?
It helps but has limits. Few-shot beat zero-shot, yet prompting alone never closed the gap to a supervised baseline. Fine-tuning drove the largest gains.
Should I still use AI for qualitative coding?
Yes, with guardrails: use it for a first-pass draft, validate per-category performance, quote verbatims, and keep a human coder in the loop for final decisions.
Last updated: July 16, 2026.
\nThis is an independent editorial summary of third-party research. QualiTaTi is not affiliated with the study's authors. Primary source: von der Heyde, L., Haensch, A.-C., Weiß, B., & Daikeler, J. (2025). \"AIn't Nothing But a Survey? Using Large Language Models for Coding German Open-Ended Survey Responses on Survey Motivation.\" Survey Research Methods (arXiv:2506.14634).
\n","related":[{"slug":"ai-conversational-interviewing-2026-study","title":"Can AI Conduct Research Interviews? A 2026 Field Study","description":"Wuttke et al. (2026) ran AI-led interviews with 571 respondents on migration attitudes: richer reasoning than surveys, and ratings at or above the survey. Delivery mode matters.","category":"Guides","date":"2026-07-31","readTime":"8 min read","author":"Qualitati Research Team"},{"slug":"digital-twins-survey-respondents-2026-study","title":"Can Digital Twins Replace Survey Respondents? A 2026 Study","description":"A 2026 Columbia Business School study found LLM digital twins correlate with real survey answers at only r = 0.20 across 164 outcomes: funhouse mirrors, not substitutes.","category":"Guides","date":"2026-07-29","readTime":"8 min read","author":"Qualitati Research Team"},{"slug":"is-ai-qualitative-coding-biased-2025","title":"Is AI Qualitative Coding Biased? What a 2025 Study Found","description":"A 2025 Sociological Methods & Research study found ChatGPT and Llama 2 coding errors correlated with respondent traits; a small supervised model was more accurate and less biased.","category":"Guides","date":"2026-07-28","readTime":"8 min read","author":"Qualitati Research Team"}]}