A Comparison of the Quality of Data-Driven Programming Hint Generation Algorithms

Price TW, Dong Y, Zhi R, Paaßen B, Lytle N, Cateté V, Barnes T (2019)
International Journal of Artificial Intelligence in Education 29(3): 368-395.

Zeitschriftenaufsatz | E-Veröff. vor dem Druck | Englisch
 
Download
Es wurde kein Volltext hochgeladen. Nur Publikationsnachweis!
Autor*in
Price, Thomas W.; Dong, Yihuan; Zhi, Rui; Paaßen, BenjaminUniBi ; Lytle, Nicholas; Cateté, Veronica; Barnes, Tiffany
Abstract / Bemerkung
In the domain of programming, a growing number of algorithms automatically generate data-driven, next-step hints that suggest how students should edit their code to resolve errors and make progress. While these hints have the potential to improve learning if done well, few evaluations have directly assessed or compared the quality of different hint generation approaches. In this work, we present the QualityScore procedure, a novel method for automatically evaluating and comparing the quality of next-step programming hints using expert ratings. We first demonstrate that the automated QualityScore ratings agree with experts’ manual ratings. We then use the QualityScore procedure to compare the quality of six data-driven, next-step hint generation algorithms using two distinct programming datasets in two different programming languages. Our results show that there are large and significant differences between the quality of the six algorithms and that these differences are relatively consistent across datasets and problems. We also identify situations where the six algorithms struggle to produce high-quality hints, and we suggest ways that future work might address these challenges. We make our methods and data publicly available and encourage researchers to use the QualityScore procedure to evaluate additional algorithms and benchmark them against our results.
Stichworte
Data-driven hints; Programming; Intelligent tutoring systems; Hint quality
Erscheinungsjahr
2019
Zeitschriftentitel
International Journal of Artificial Intelligence in Education
Band
29
Ausgabe
3
Seite(n)
368-395
ISSN
1560-4292
eISSN
1560-4306
Page URI
https://pub.uni-bielefeld.de/record/2935953

Zitieren

Price TW, Dong Y, Zhi R, et al. A Comparison of the Quality of Data-Driven Programming Hint Generation Algorithms. International Journal of Artificial Intelligence in Education. 2019;29(3):368-395.
Price, T. W., Dong, Y., Zhi, R., Paaßen, B., Lytle, N., Cateté, V., & Barnes, T. (2019). A Comparison of the Quality of Data-Driven Programming Hint Generation Algorithms. International Journal of Artificial Intelligence in Education, 29(3), 368-395. doi:10.1007/s40593-019-00177-z
Price, T. W., Dong, Y., Zhi, R., Paaßen, B., Lytle, N., Cateté, V., and Barnes, T. (2019). A Comparison of the Quality of Data-Driven Programming Hint Generation Algorithms. International Journal of Artificial Intelligence in Education 29, 368-395.
Price, T.W., et al., 2019. A Comparison of the Quality of Data-Driven Programming Hint Generation Algorithms. International Journal of Artificial Intelligence in Education, 29(3), p 368-395.
T.W. Price, et al., “A Comparison of the Quality of Data-Driven Programming Hint Generation Algorithms”, International Journal of Artificial Intelligence in Education, vol. 29, 2019, pp. 368-395.
Price, T.W., Dong, Y., Zhi, R., Paaßen, B., Lytle, N., Cateté, V., Barnes, T.: A Comparison of the Quality of Data-Driven Programming Hint Generation Algorithms. International Journal of Artificial Intelligence in Education. 29, 368-395 (2019).
Price, Thomas W., Dong, Yihuan, Zhi, Rui, Paaßen, Benjamin, Lytle, Nicholas, Cateté, Veronica, and Barnes, Tiffany. “A Comparison of the Quality of Data-Driven Programming Hint Generation Algorithms”. International Journal of Artificial Intelligence in Education 29.3 (2019): 368-395.