Document Type

Conference Paper

Journal/Book Title/Conference

2025 Annual Meeting of the American Educational Research Association (AERA)

Location

Denver, CO

Publication Date

4-2025

First Page

1

Last Page

16

Abstract

As an integral part of qualitative research inquiry, field notes provide important data from researchers embedded in research sites. However, field notes can vary significantly, influenced by the researchers' immersion in the field, prior knowledge, beliefs, interests, and perspectives. As consequence, their interpretation presents significant challenges. This study offers a preliminary investigation into the potential of using large language models to assist researchers with the analysis and interpretation of field notes data. Our methodology consisted of two phases. First, a researcher deductively coded field notes of six classroom implementations of a novel elementary-level mathematics curriculum. In the second phase, we prompted ChatGPT-4 to code the same field notes, using the codebook, definitions, examples, and deductive coding approach employed by the researcher. We also prompted Chatgpt to provide justifications of its coding decisions We then, calculated agreements and disagreements between ChatGPT and the researcher, organized the data in a contingency table, computed Cohen's Kappa, structured the data into a confusion matrix; and using the researcher’s coding as the “gold standard”, we calculated performance measures, specifically: Accuracy, Precision, Recall, and F1 Score. Our findings revealed that while the researcher and ChatGPT appeared to generally agree on the frequency in applying the different codes, overall agreement, as measured by Cohen’s Kappa was low. In contrast, using measures from information science at the code level revealed more nuanced results. Moreover, coupled with ChatGPT justifications of coding decisions, these findings provided insights than can help support the iterative improvement of codebooks.

Comments

This work was supported by National Science Foundation Grant no. 2031382 and 2031404. Opinions, findings, or recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of the funding agency. We thank the participating teachers and students.

Download

Additional Files

ITLSstures2025-LopezFierroShehzadZandi-StreamliningFieldNote-powerpoint (1).pdf (5704 kB)

Share

COinS