Inductive Coding with ChatGPT - An Evaluation of Different GPT Models Clustering Qualitative Data into Categories
Arlinghaus CS, Wulff C, Maier GW (2024)
OSF Preprints.
Preprint | Englisch
Download
Es wurden keine Dateien hochgeladen. Nur Publikationsnachweis!
Autor*in
Abstract / Bemerkung
Qualitative data is invaluable, yet its analysis is very time-consuming. To prevent the loss of valuable information and to streamline the coding process for developing and assigning inductive categories, we introduce LLM-Assisted Inductive Categorization (LAIC), a novel method of categorizing text responses using a Large Language Model (LLM). In two pre-registered studies, we tested two Generative Pre-trained Transformer (GPT) models that are commonly used in ChatGPT (GPT-3.5 Turbo and GPT-4o) across three temperature settings (0, 0.5, 1) with 10 repetitions each (120 runs in total). Outputs were evaluated based on established qualitative research criteria (credibility, dependability, confirmability, transferability, transparency). Two human coders also generated inductive categories and assigned text responses accordingly for comparison. Our findings demonstrate that both GPT models are highly effective in developing and assigning inductive categories, even outperforming human coders in agreement rates. Overall, GPT-4o achieved the best results (e.g., better explanations and higher agreement) and is recommended for inductive category formation and assignment with a temperature setting of 0 and 10 repetitions. This approach saves significant time and resources while enhancing analysis quality. Instructions and Python scripts for applying our new coding technique are freely available under a CC-BY 4.0 International license: https://osf.io/h4dux/
Stichworte
ChatGPT;
GPT models;
inductive coding;
inductive category formation;
qualitative research;
new AI method
Erscheinungsjahr
2024
Zeitschriftentitel
OSF Preprints
Page URI
https://pub.uni-bielefeld.de/record/2993492
Zitieren
Arlinghaus CS, Wulff C, Maier GW. Inductive Coding with ChatGPT - An Evaluation of Different GPT Models Clustering Qualitative Data into Categories. OSF Preprints. 2024.
Arlinghaus, C. S., Wulff, C., & Maier, G. W. (2024). Inductive Coding with ChatGPT - An Evaluation of Different GPT Models Clustering Qualitative Data into Categories. OSF Preprints. https://doi.org/10.31219/osf.io/gpnye
Arlinghaus, Clarissa Sabrina, Wulff, Charlotte, and Maier, Günter W. 2024. “Inductive Coding with ChatGPT - An Evaluation of Different GPT Models Clustering Qualitative Data into Categories”. OSF Preprints.
Arlinghaus, C. S., Wulff, C., and Maier, G. W. (2024). Inductive Coding with ChatGPT - An Evaluation of Different GPT Models Clustering Qualitative Data into Categories. OSF Preprints.
Arlinghaus, C.S., Wulff, C., & Maier, G.W., 2024. Inductive Coding with ChatGPT - An Evaluation of Different GPT Models Clustering Qualitative Data into Categories. OSF Preprints.
C.S. Arlinghaus, C. Wulff, and G.W. Maier, “Inductive Coding with ChatGPT - An Evaluation of Different GPT Models Clustering Qualitative Data into Categories”, OSF Preprints, 2024.
Arlinghaus, C.S., Wulff, C., Maier, G.W.: Inductive Coding with ChatGPT - An Evaluation of Different GPT Models Clustering Qualitative Data into Categories. OSF Preprints. (2024).
Arlinghaus, Clarissa Sabrina, Wulff, Charlotte, and Maier, Günter W. “Inductive Coding with ChatGPT - An Evaluation of Different GPT Models Clustering Qualitative Data into Categories”. OSF Preprints (2024).