Reading Times Predict the Quality of Generated Text Above and Beyond Human Ratings

Zarrieß S, Loth S, Schlangen D (2015)
In: Proceedings of the 15th European Workshop on Natural Language Generation. Belz A, Gatt A, Portet F, Purver M (Eds);BRIGHTON, East Sussex, United Kingdom: The Association for Computational Linguistics: 38-47.

Conference Paper | Published | English

No fulltext has been uploaded

Editor
Belz, Anya ; Gatt, Albert ; Portet, François ; Purver, Matthew
Abstract
Typically, human evaluation of NLG output is based on user ratings. We collected ratings and reading time data in a simple, low-cost experimental paradigm for text generation. Participants were presented corpus texts, automatically linearised texts, and texts containing predicted referring expressions and automatic linearisation. We demonstrate that the reading time metrics outperform the ratings in classifying texts according to their quality. Regression analyses showed that self-reported ratings discriminated poorly between the kinds of manipulation, especially between defects in word order and text coherence. In contrast, a combination of objective measures from the low-cost mouse contingent reading paradigm provided very high classification accuracy and thus, greater insight into the actual quality of an automatically generated text.
Publishing Year
Conference
15th European Workshop on Natural Language Generation
Location
BRIGHTON, Sussex, United Kingdom
Conference Date
2015-09-10 – 2015-09-11
PUB-ID

Cite this

Zarrieß S, Loth S, Schlangen D. Reading Times Predict the Quality of Generated Text Above and Beyond Human Ratings. In: Belz A, Gatt A, Portet F, Purver M, eds. Proceedings of the 15th European Workshop on Natural Language Generation. BRIGHTON, East Sussex, United Kingdom: The Association for Computational Linguistics; 2015: 38-47.
Zarrieß, S., Loth, S., & Schlangen, D. (2015). Reading Times Predict the Quality of Generated Text Above and Beyond Human Ratings. In A. Belz, A. Gatt, F. Portet, & M. Purver (Eds.), Proceedings of the 15th European Workshop on Natural Language Generation (pp. 38-47). BRIGHTON, East Sussex, United Kingdom: The Association for Computational Linguistics.
Zarrieß, S., Loth, S., and Schlangen, D. (2015). “Reading Times Predict the Quality of Generated Text Above and Beyond Human Ratings” in Proceedings of the 15th European Workshop on Natural Language Generation, ed. A. Belz, A. Gatt, F. Portet, and M. Purver (BRIGHTON, East Sussex, United Kingdom: The Association for Computational Linguistics), 38-47.
Zarrieß, S., Loth, S., & Schlangen, D., 2015. Reading Times Predict the Quality of Generated Text Above and Beyond Human Ratings. In A. Belz, et al., eds. Proceedings of the 15th European Workshop on Natural Language Generation. BRIGHTON, East Sussex, United Kingdom: The Association for Computational Linguistics, pp. 38-47.
S. Zarrieß, S. Loth, and D. Schlangen, “Reading Times Predict the Quality of Generated Text Above and Beyond Human Ratings”, Proceedings of the 15th European Workshop on Natural Language Generation, A. Belz, et al., eds., BRIGHTON, East Sussex, United Kingdom: The Association for Computational Linguistics, 2015, pp.38-47.
Zarrieß, S., Loth, S., Schlangen, D.: Reading Times Predict the Quality of Generated Text Above and Beyond Human Ratings. In: Belz, A., Gatt, A., Portet, F., and Purver, M. (eds.) Proceedings of the 15th European Workshop on Natural Language Generation. p. 38-47. The Association for Computational Linguistics, BRIGHTON, East Sussex, United Kingdom (2015).
Zarrieß, Sina, Loth, Sebastian, and Schlangen, David. “Reading Times Predict the Quality of Generated Text Above and Beyond Human Ratings”. Proceedings of the 15th European Workshop on Natural Language Generation. Ed. Anya Belz, Albert Gatt, François Portet, and Matthew Purver. BRIGHTON, East Sussex, United Kingdom: The Association for Computational Linguistics, 2015. 38-47.
This data publication is cited in the following publications:
This publication cites the following data publications:

Export

0 Marked Publications

Open Data PUB

Search this title in

Google Scholar
ISBN Search