Reading Times Predict the Quality of Generated Text Above and Beyond Human Ratings

Zarrieß S, Loth S, Schlangen D (2015)
In: Proceedings of the 15th European Workshop on Natural Language Generation. Belz A, Gatt A, Portet F, Purver M (Eds); BRIGHTON, East Sussex, United Kingdom: The Association for Computational Linguistics: 38-47.

Download
Es wurde kein Volltext hochgeladen. Nur Publikationsnachweis!
Konferenzbeitrag | Veröffentlicht | Englisch
Herausgeber
; ; ;
Abstract / Bemerkung
Typically, human evaluation of NLG output is based on user ratings. We collected ratings and reading time data in a simple, low-cost experimental paradigm for text generation. Participants were presented corpus texts, automatically linearised texts, and texts containing predicted referring expressions and automatic linearisation. We demonstrate that the reading time metrics outperform the ratings in classifying texts according to their quality. Regression analyses showed that self-reported ratings discriminated poorly between the kinds of manipulation, especially between defects in word order and text coherence. In contrast, a combination of objective measures from the low-cost mouse contingent reading paradigm provided very high classification accuracy and thus, greater insight into the actual quality of an automatically generated text.
Erscheinungsjahr
Titel des Konferenzbandes
Proceedings of the 15th European Workshop on Natural Language Generation
Seite
38-47
Konferenz
15th European Workshop on Natural Language Generation
Konferenzort
BRIGHTON, Sussex, United Kingdom
Konferenzdatum
2015-09-10 – 2015-09-11
PUB-ID

Zitieren

Zarrieß S, Loth S, Schlangen D. Reading Times Predict the Quality of Generated Text Above and Beyond Human Ratings. In: Belz A, Gatt A, Portet F, Purver M, eds. Proceedings of the 15th European Workshop on Natural Language Generation. BRIGHTON, East Sussex, United Kingdom: The Association for Computational Linguistics; 2015: 38-47.
Zarrieß, S., Loth, S., & Schlangen, D. (2015). Reading Times Predict the Quality of Generated Text Above and Beyond Human Ratings. In A. Belz, A. Gatt, F. Portet, & M. Purver (Eds.), Proceedings of the 15th European Workshop on Natural Language Generation (pp. 38-47). BRIGHTON, East Sussex, United Kingdom: The Association for Computational Linguistics.
Zarrieß, S., Loth, S., and Schlangen, D. (2015). “Reading Times Predict the Quality of Generated Text Above and Beyond Human Ratings” in Proceedings of the 15th European Workshop on Natural Language Generation, Belz, A., Gatt, A., Portet, F., and Purver, M. eds. (BRIGHTON, East Sussex, United Kingdom: The Association for Computational Linguistics), 38-47.
Zarrieß, S., Loth, S., & Schlangen, D., 2015. Reading Times Predict the Quality of Generated Text Above and Beyond Human Ratings. In A. Belz, et al., eds. Proceedings of the 15th European Workshop on Natural Language Generation. BRIGHTON, East Sussex, United Kingdom: The Association for Computational Linguistics, pp. 38-47.
S. Zarrieß, S. Loth, and D. Schlangen, “Reading Times Predict the Quality of Generated Text Above and Beyond Human Ratings”, Proceedings of the 15th European Workshop on Natural Language Generation, A. Belz, et al., eds., BRIGHTON, East Sussex, United Kingdom: The Association for Computational Linguistics, 2015, pp.38-47.
Zarrieß, S., Loth, S., Schlangen, D.: Reading Times Predict the Quality of Generated Text Above and Beyond Human Ratings. In: Belz, A., Gatt, A., Portet, F., and Purver, M. (eds.) Proceedings of the 15th European Workshop on Natural Language Generation. p. 38-47. The Association for Computational Linguistics, BRIGHTON, East Sussex, United Kingdom (2015).
Zarrieß, Sina, Loth, Sebastian, and Schlangen, David. “Reading Times Predict the Quality of Generated Text Above and Beyond Human Ratings”. Proceedings of the 15th European Workshop on Natural Language Generation. Ed. Anya Belz, Albert Gatt, François Portet, and Matthew Purver. BRIGHTON, East Sussex, United Kingdom: The Association for Computational Linguistics, 2015. 38-47.
Link(s) zu Volltext(en)
Access Level
Restricted Closed Access

Export

Markieren/ Markierung löschen
Markierte Publikationen

Open Data PUB

Suchen in

Google Scholar
ISBN Suche