Can open-source LLMs running on your own device code qualitative data privately? A 2026 CHI study testing ChatQDA — an open-source, on-device coding tool — found researchers will use local LLMs for basic extraction, but "conditional trust" and a privacy-perception gap mean technical security alone does not convince them their interview data is safe.
What the study tested
The study evaluated whether a privacy-preserving, on-device large language model can support qualitative open coding without sending sensitive transcripts to commercial cloud services. According to Ngo, Nguyen Van, Nguyen, Do, and Nguyen-Quoc (2026), the team built ChatQDA, a framework powered by the open-source gpt-oss-20b model (a 20.9-billion-parameter mixture-of-experts model chosen for a compact, on-device hardware footprint), and ran a mixed-methods user study comparing manual open coding against ChatQDA-assisted coding on a five-page semi-structured interview transcript.
The sample was small and deliberately mixed: four participants, split between two novice qualitative researchers (fewer than two years of experience) and two experienced researchers (two or more years). This is an early-stage, formative study — its value is in the patterns it surfaces, not in generalizable effect sizes.
Why on-device LLMs matter for qualitative research
Most AI coding tools send transcripts to external servers. For interview data that may contain health details, identifiable quotes, or IRB-restricted material, that transfer raises real concerns: data sovereignty, re-identification risk, and institutional-review-board compliance. An on-device model keeps the data on the researcher's own machine — in principle removing the transfer entirely. The study's contribution is testing whether researchers actually believe that, and whether the coding output is good enough to rely on.
What did researchers think of AI-assisted coding?
Participants found the tool easy to learn but were cautious about trusting its interpretations. On a 5-point Likert scale reported by the authors, the system scored well on usability and speed but only moderately on trust and depth of interpretation.
| Dimension (5-point scale) | Mean rating | Read |
|---|---|---|
| Ease of learning & clarity | 4.0 | Strong — quick to pick up |
| Task completion speed | 3.75 (50% rated 5) | Fast for extraction |
| Overall usefulness | 3.00 | Moderate |
| Overall trustworthiness | 3.00 | Conditional |
| Reliability | 3.00 | Doubted for interpretation |
| General security measures (perceived) | 2.75 | Weakest score |
According to the authors, participants showed "conditional trust": they valued ChatQDA for basic extraction tasks — pulling out candidate codes and segments — but doubted its interpretive accuracy and reliability for the parts of coding that require human judgment.
The privacy-perception gap: secure but not trusted
The most striking finding is what the authors call a privacy-perception gap. Even though the model ran locally — technically keeping data off external servers — perceived security was the lowest-rated dimension (M=2.75 for general security measures, M=3.0 for protection against third-party interception). Participants reported "epistemic uncertainty regarding data protection": because the security is invisible, it did not translate into confidence.
The authors' core recommendation follows directly: privacy-preserving analysis tools must prioritize verifiable privacy and methodological rigor, not just technical infrastructure. Telling researchers "it runs locally" is not enough; they need to be able to see and confirm where their data goes.
What this means for researchers
Three practical takeaways:
- Use AI for extraction, keep interpretation human. The conditional-trust pattern matches a broader theme in 2026 AI-coding research: LLMs are useful for first-pass code generation and surfacing missed segments, but theme interpretation stays with the researcher. A human-in-the-loop thematic analysis workflow reflects exactly this division of labor.
- Privacy claims need to be auditable. If a tool handles sensitive transcripts, "trust us, it's local/encrypted" will not satisfy careful researchers or IRBs. Look for verifiable data-handling documentation.
- Treat small studies as signals, not proof. With n=4, these results are directional. They flag a real design problem — the gap between actual and perceived security — rather than measuring how big it is.
For teams weighing where AI fits in their coding pipeline, the lesson is to scope it to mechanical, reviewable steps and route interpretation through a transparent QDA workspace where every AI-suggested code can be checked against the source.
FAQ
Can open-source LLMs run qualitative coding offline?
Yes. The ChatQDA study used gpt-oss-20b on-device, with no cloud transfer of transcripts, to perform open coding on interview data.
Are on-device LLMs more private than cloud AI for research data?
Technically yes — local models keep data off external servers. But the 2026 study found researchers still felt uncertain about data protection, so verifiable privacy matters as much as the architecture itself.
Should I trust AI to interpret my qualitative data?
Not unsupervised. Participants in the study trusted AI for basic extraction but doubted its interpretive reliability. Keep theme interpretation and final coding decisions with a human researcher.
How big was the study?
Four participants — two novice and two experienced qualitative researchers — coding a five-page transcript. It is a formative user study, so treat findings as design signals rather than generalizable benchmarks.
The bottom line
On-device, open-source LLMs make private qualitative coding technically feasible today. But the 2026 ChatQDA study shows the harder problem is trust: researchers will let AI extract, not interpret, and they need privacy they can verify, not just privacy they are promised.
Primary source: Ngo, T. T., Nguyen Van, D., Nguyen, A.-M., Do, P.-A., & Nguyen-Quoc, A. (2026). Qualitative Coding Analysis through Open-Source Large Language Models: A User Study and Design Recommendations. Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems. arxiv.org/abs/2602.18352
Last updated: June 1, 2026.
This is an independent editorial summary of third-party research, written to help researchers understand the study. It is not affiliated with or endorsed by the original authors.