AI you can defend in peer review
Most AI research tools ask you to trust them. Qualitati measures its AI, publishes the results with the methodology, and builds verification into the product — every quote checked against its transcript, every code tied to its exact text span, reliability reported in the statistics reviewers actually expect.
Last updated: July 12, 2026. All results below are from internal evaluations; each is dated and described so it can be challenged.
Our verification principle: verify or drop
An AI output that cannot be anchored to your source data is removed, never silently invented. ThemeLens drops quotes it cannot locate in a transcript. The QDA Workspace resolves every AI code to an exact character span or discards it. Voice Analytics marks uncalibratable emotion labels as low-confidence instead of guessing. Digital twins report when they lack grounding data for a question. This principle costs us impressive-looking output; it buys you findings you can defend to reviewers, clients, and regulators.
Published evaluation results (as of July 12, 2026)
| Tool | What we measure | Result |
|---|---|---|
| AI Interviewer | Leading questions | 0% Across all scored turns in a July 2026 internal evaluation with English and Chinese participant personas, the AI interviewer asked no leading questions — questions that presuppose or suggest a desired answer. Method: Simulated interviews against the production prompt pipeline; each turn scored by an LLM judge with rule-based checks, spot-checked by a researcher. |
| AI Interviewer | Probing on shallow answers | 83% When a participant gave a shallow or one-line answer, the interviewer followed up with a probe (rather than moving on) in 83% of opportunities. Method: Same July 2026 evaluation; probe opportunities identified from turn context, follow-up behaviour scored per opportunity. |
| ThemeLens (thematic analysis) | Quote verification | 100% of displayed quotes verified Every supporting quote shown in a ThemeLens report is checked against the source transcript before display. Quotes that cannot be located are dropped; quotes attributed to the wrong participant are reassigned. Nothing unverifiable is shown. Method: A verification pass runs inside the analysis pipeline itself (normalized text matching, ellipsis-aware). Kept / reassigned / dropped counts are recorded for every analysis. |
| QDA Workspace (AI coding) | Segment-offset fidelity | 100% On our ground-truth coding benchmark, every AI-coded segment now maps to the exact character span in the source document — up from 0% before the July 2026 offset-resolution work. Codes point at the text they actually describe. Method: Fixture transcripts with known correct segments; automated comparison of returned offsets against ground truth. |
| QDA Workspace (reliability) | Inter-rater reliability | Cohen's kappa, code-level When two independent AI coders (different model families) code the same document, agreement is reported with code-level Cohen's kappa — the same statistic reviewers expect from human double-coding — instead of a vague match percentage. Method: Segment pairs matched by span overlap; kappa computed on code assignment; coverage-based kappa reported when segmentation differs. |
| Annotator (batch coding) | Inter-coder reliability on every job | Krippendorff's α, Fleiss' κ, Cohen's κ Every multi-model annotation job returns reliability statistics across the models that coded it, with per-column agreement heatmaps, so you can see exactly where models disagree before trusting a column. Method: Standard reliability estimators computed over the models' parallel codings of identical rows. |
| Voice Analytics | Honest confidence labelling | Per-speaker calibration Emotion labels are computed relative to each speaker's own vocal baseline rather than absolute thresholds, and labels that cannot be calibrated (too few turns, flat delivery) are explicitly marked low-confidence instead of over-claimed. Method: Robust per-speaker baselines (median/IQR z-scores); emotion-bearing words in the transcript anchor vocal expressiveness; text–voice incongruence is flagged for human re-listening. |
| Digital Twin Panel | Grounded answers + enforced consent | Verbatim grounding Twin answers are grounded in the participant's actual interview turns retrieved for each question, calibration honestly reports when source data is insufficient, and twins cannot be activated or queried without explicit participant consent — enforced server-side and audited. Method: Retrieval over source transcripts (English and Chinese); calibration status computed from grounding coverage; consent checks at activation and at every simulated interaction. |
How we evaluate
Evaluations run against the same production code that serves users, not simplified stand-ins. Ground-truth benchmarks use fixture transcripts with known correct answers; interviewer behaviour is scored per turn by an LLM judge constrained by explicit rules, with researcher spot-checks. Results become regression tests: a code change that degrades quote verification, offset fidelity, or interviewer behaviour fails our test suite before it ships. Evaluations are re-run when the underlying prompts, models, or pipelines change, and this page is updated with the new numbers — whichever direction they move.
What we don't claim
These are internal evaluations, not third-party audits. Evaluation sets are modest in size and grow over time. A 0% leading-question rate in our benchmark does not guarantee 0% in every live interview — large language models vary between runs, which is exactly why the product reports reliability statistics on your own data instead of asking you to extrapolate from ours. We recommend human validation of AI coding for any publication-bound analysis, and the platform is built to make that validation fast rather than to replace it.
Frequently asked questions
Is AI qualitative coding accurate enough to use in research?
Used with verification, yes. Qualitati's position is that AI coding must be validated the same way a second human coder would be: report inter-rater reliability (Cohen's kappa) against a human-coded sample, audit disagreements, and keep a human in the validation loop. The platform computes these statistics for you rather than asking you to trust the AI.
How does Qualitati prevent hallucinated quotes in thematic analysis?
Every quote in a ThemeLens report is verified against the source transcript before it is displayed. Quotes that cannot be found are dropped, and quotes attributed to the wrong participant are reassigned to the correct one. This verify-or-drop rule means a quote you see in a report exists in your data.
Can I verify the AI's work myself?
Yes. The QDA Workspace links every AI code to the exact text span it describes, ThemeLens quotes carry participant attribution you can trace to the transcript, the Annotator returns per-row model outputs with reliability statistics, and interview transcripts and recordings are always downloadable for independent audit.
Are these numbers independently audited?
Not yet — they come from internal evaluations, and we say so. Each number on this page is dated, the evaluation method is described next to it, and the evaluation harnesses run as regression tests so a change that degrades a result is caught before release. Third-party benchmarking is on our roadmap, and we publish the methodology precisely so others can challenge it.
How often is this page updated?
Whenever an evaluation is re-run or a verification mechanism changes — the "Last updated" date at the top reflects the most recent revision. Numbers are never carried forward silently: each result is tied to the evaluation that produced it.