Leveraging Consistency: Evaluating AI Coders for Qualitative Research

Authors

• Saríah López-Fierro, Utah State UniversityFollow
• Carlos Chiriboga-Calderón, Soluciones Wandarina S. A.Follow
• José López-Fierro, Independent ResearcherFollow
• Rubén Pacheco-Villamar, Universidad Espíritu SantoFollow

Document Type

Article

Journal/Book Title/Conference

Decision Sciences Institute (DSI) 2025 Annual Conference (in Orlando)

Publisher

Decision Sciences Institute

Location

Orlando, FL

Publication Date

11-22-2025

Journal Article Version

Accepted Manuscript

First Page

1

Last Page

13

Creative Commons License

This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 License.

Abstract

This study provides insights of using Large Language Models (LLMs) for deductive coding tasks in market research. Deductive coding, which applies predefined codes to text, often demands consistency rather than interpretive nuance. 

We compare five closed-weight LLMs with a group of human coders in tagging instances of “Expansive Framing” across qualitative excerpts, using Fleiss’ Kappa, ANOVA, and Estimated Marginal Means. Results reveal that LLMs exhibit higher consistency and faster processing than humans. These findings suggest that integrating LLMs can improve efficiency, making them an attractive asset for firms managing large datasets.

Recommended Citation

López-Fierro, Saríah; Chiriboga-Calderón, Carlos; López-Fierro, José; and Pacheco-Villamar, Rubén, "Leveraging Consistency: Evaluating AI Coders for Qualitative Research" (2025). Instructional Technology and Learning Sciences Faculty Publications. Paper 915.
https://digitalcommons.usu.edu/itls_facpub/915