Short answer: Not by default. Large language models can return different outputs for identical inputs — even at temperature 0 — and hosted models change under your feet through silent version updates and deprecations. For qualitative researchers this threatens dependability and confirmability. Reproducibility is achievable in a documented, statistical form: log your full configuration, run the analysis at least twice, report test-retest agreement (Cohen's kappa between runs), and validate against a human-coded sample.
\n\nReproducibility in qualitative analysis has always been a nuanced claim — two human coders never agree perfectly either. But when an LLM does the coding, two distinct instabilities enter the pipeline that most methods sections currently ignore. The first is within-version non-determinism: the same model, same prompt, and same settings can produce different codes on Tuesday than on Monday. The second is across-version drift: the model you cite in your paper may behave differently — or no longer exist — by the time a reviewer or replicator tries to re-run your analysis.
\nLast updated: July 12, 2026. This guide covers why both instabilities happen, what the empirical evidence says, why it matters for qualitative rigor, and a concrete logging-and-agreement protocol you can adopt today.
\n\nSampling temperature is the obvious source of randomness: at typical settings the model draws tokens probabilistically, so two runs diverge by design. Most researchers know to set temperature to 0 (greedy decoding) for analytical tasks. Fewer know that this does not produce identical outputs either.
\nOuyang and colleagues systematically re-ran identical code-generation requests and found substantial instability even under “deterministic” settings; under default settings, the majority of tasks produced zero test-equivalent outputs across repeated requests (ACM TOSEM, 2025). A separate study of supposedly deterministic configurations across multiple models reached the same conclusion: greedy decoding narrows but does not close the gap (Atil et al., 2024).
\nThe mechanism is now well understood. In one demonstration, sampling 1,000 completions of the same prompt at temperature 0 produced 80 unique outputs. The dominant cause is not floating-point noise alone but batch-size dependence: the numerical result of a GPU reduction depends on how many other requests are batched with yours on the server, which you cannot see or control as an API user (Thinking Machines Lab, 2025). Mixture-of-Experts architectures add another path: expert routing can shift with server load. When two candidate tokens are nearly tied, a one-bit difference in an intermediate activation flips the output — and every token after it.
\nThe practical upshot for a researcher: you cannot promise bit-identical re-runs of a hosted LLM analysis. What you can promise is a documented configuration and a measured level of run-to-run agreement.
\n\nHosted models are moving targets. Providers retrain, fine-tune, and silently update the models behind stable-sounding names. The canonical demonstration is Chen, Zaharia and Zou's longitudinal evaluation: between March and June 2023, GPT-4's accuracy at identifying prime numbers dropped from 84% to 51%, code formatting errors increased, and willingness to follow user instructions declined — while the same period improved other tasks (How Is ChatGPT's Behavior Changing over Time?, 2023). The trends for GPT-3.5 and GPT-4 often pointed in opposite directions, which means you cannot even assume drift is monotonic improvement.
\nDeprecation is the harder deadline. Providers retire model snapshots on schedules of months to a few years. Once the snapshot you used is gone, exact replication of your analysis becomes impossible — no amount of logging brings the instrument back. Two mitigations exist: pin dated snapshots rather than floating aliases (for example, a versioned model ID rather than a “latest” alias), and archive the model's raw outputs alongside your data so the analysis itself remains auditable after the model is retired.
\n\nIn Lincoln and Guba's trustworthiness framework, dependability asks whether findings are consistent and could be repeated, and confirmability asks whether findings are shaped by the data rather than by analyst (or instrument) idiosyncrasy. Both are conventionally demonstrated through an audit trail: a reader should be able to trace every theme back through coding decisions to raw data.
\nAn unstable AI coder undermines both at once. If the “coder” produces different codes on re-run, the coding decisions in your audit trail describe one realization of a random process, not a stable procedure. And the evidence says the instability is large enough to matter for research: Reiss found that repeating identical annotation inputs to ChatGPT produced consistency below accepted reliability thresholds, and that even minor prompt rewording shifted outputs (Reiss, 2023). Task type matters too: in a medical-education coding study, GPT-4 applying a fixed human codebook reached a mean Cohen's kappa of 0.71 against humans, but in inductive mode only 31% of AI-generated codes matched human ones (Journal of Learning Analytics, 2024). Deductive coding with a tight codebook is far more stable ground than open inductive generation.
\n\nTreat the model as an instrument and log its calibration. The following table is the minimum a methods section (or supplementary file) should contain.
\n| What to log | Why it matters |
|---|---|
| Provider and exact model ID (pinned, dated snapshot) | Floating aliases silently change; a dated snapshot is the closest thing to a fixed instrument |
| Access date(s) of every analysis run | Anchors the analysis against version drift; providers update models without notice |
| Sampling parameters (temperature, top_p, max tokens, seed if offered) | These change output distributions dramatically; “defaults” differ across providers and over time |
| Full prompt text: system prompt, task instructions, codebook, few-shot examples | Minor wording changes measurably shift annotations (Reiss, 2023) |
| Number of runs and the aggregation rule (single run, majority vote, pooling) | Pooled or majority-voted outputs are more reliable than single runs and must be described to be replicated |
| Post-processing rules (code merging, normalization, exclusions) | Silent cleanup steps are a classic source of irreproducibility |
| Raw model outputs, archived | The only guaranteed replication artifact once a snapshot is deprecated |
| Between-run agreement statistic | Quantifies the instability instead of hiding it (see next section) |
Borrow test-retest logic from psychometrics: run the identical analysis twice (or more), treat the runs as two coders, and compute an agreement coefficient — Cohen's kappa for two runs on categorical codes, Krippendorff's alpha for more runs or more complex data. This turns “the model might vary” into a number a reviewer can evaluate. By the common Landis-and-Koch reading, between-run kappa above 0.80 is strong; 0.61–0.80 is substantial but worth tightening; below that, your codebook or prompt is underspecified and the single-run analysis should not be trusted.
\nThree practical refinements:
\nTest-retest agreement guards against instability; it does not guard against a model that is stably wrong. The complementary control is human review of a stratified sample: draw a random sample of AI-coded segments (stratified by code, so rare codes are checked too), have a researcher code them blind, compute human–AI kappa, and adjudicate disagreements. Keep the disagreement log — it is the confirmability evidence reviewers actually want. Run a deliberate disconfirmation pass as well: search for segments that contradict the emerging themes before accepting the AI's synthesis. Our checklist for auditing AI qualitative coding walks through this end to end.
\n\nQualitati's analysis tools are built around the assumption that AI output must be auditable rather than trusted. ThemeLens anchors every synthesized theme to verbatim participant quotes that are verified against the source transcripts, so a human can always trace a claim back to data — the confirmability requirement, operationalized. The QDA Workspace keeps AI-assisted coding under human override, letting you re-run, compare, and correct code assignments instead of accepting a single stochastic pass. And because runs are cheap, the run-twice-and-compare protocol described above is practical rather than aspirational. Pricing is transparent and pay-as-you-go, with a free tier to test the workflow.
\n\nNo. It removes sampling randomness but not the batch-dependent numerical variation in server-side inference or Mixture-of-Experts routing effects. Empirical studies find repeated identical requests still diverge at temperature 0 (Ouyang et al., 2025).
\nNo. “We used GPT-4” describes a family of changing systems, not an instrument. Report the dated snapshot ID, access dates, sampling parameters, full prompts, number of runs, and the aggregation rule — and archive raw outputs.
\nCohen's kappa for two runs over categorical codes; Krippendorff's alpha for three or more runs, missing data, or non-nominal codes. Report it alongside human–AI agreement on a validation sample.
\nIncreasingly yes, when the reliability case is made explicitly: logged configuration, between-run agreement, human validation sample, and an audit trail from themes to quotes. Undocumented single-run AI coding is what reviewers reject.
\n\nLLM analyses are not reproducible in the bit-identical sense, and pretending otherwise is a rigor problem waiting for a reviewer to find it. They are reproducible in the sense that matters for qualitative research: as a documented procedure with measured stability. Pin the snapshot, log the configuration, run it twice, report the kappa, and keep a human in the loop. If you want a workflow where the audit trail comes built in, try the Qualitati QDA Workspace — start free, no credit card required.
","related":[{"slug":"synthetic-survey-panels-marginal-fidelity-2026","title":"Do Synthetic Survey Panels Simulate People? A 2026 Test","description":"Do synthetic survey panels simulate real respondents? A 2026 test finds matching published averages is not proof: a no-persona prompt beat panels 9 of 9 times.","category":"Methodology","date":"2026-10-22","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"llm-as-judge-qualitative-interpretation-2026","title":"Can LLM-as-Judge Rate Qualitative Interpretations? A 2026 Test","description":"Can LLM-as-judge replace human review of qualitative interpretations? A 2026 study of 3,560 AI interview readings finds it screens models but misses nuance.","category":"Methodology","date":"2026-10-21","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"rehearse-interview-protocol-simulated-participants-2026","title":"Can You Rehearse Interviews with AI Participants? A 2026 Study","description":"Can you rehearse an interview guide with AI-simulated participants? A 2026 study of 21 researchers found real protocol revisions and three hidden risks.","category":"Methodology","date":"2026-10-21","readTime":"7 min read","author":"Qualitati Research Team"}]}