PUB - Publikationen an der Universität Bielefeld

The goal of our paper is twofold. On a more practical level, it examines the differences in behaviour of book reviewers across ratings and genres. On an epistemological level, it discusses and criticises the (computational) methods used to analyse such differences. The work is based on the LOBO corpus, comprising ~1.3 million book reviews in German language, downloaded from the social reading platform LovelyBooks. Our methods of analysis fall into two main categories: wordcount-based sentiment analysis (SA) and advanced machine learning (ML). For the first category, a repository of six SA lexicons (sentiWS, NRC, ADU, LANG, Plutchik, and Ekman) was created. Lexicon formats were uniformed to automatically annotate reviews in a processing pipeline. For the second category, two ML approaches based on distributional representations of natural language (BERT and FastText) were trained on two manually-annotated datasets: in the first dataset (~21,000 sentences), the annotation task was that of identifying evaluative language (vs. descriptive language); in the second dataset (~6,000 sentences), the task focused on the distinction between positive and negative sentiment. Wordcount-based SA showed limitations in the extensiveness and coverage of lexicons (e.g., Plutchik has a hit rate of only 2.4% on the entire LOBO corpus), as well as in their reliability (detailed analysis of the German NRC lexicon identified ~28% of disputable emotion assignments). Advanced ML techniques depend primarily on the quality of the training material. However, the two annotators working on the LOBO corpus reached strong levels of agreement for both tasks (Cohen’s Kappa ~ 0.8), indicating their possible automation. The annotated dataset offered also a ground truth for comparing the two methods: overall, ML proved substantially more efficient than SA. In a 5-fold cross validation (repeated five times to average variance), BERT reached a macro F1-score of .89 for the evaluative language task and of .85 for the positive vs. negative sentiment task (FastText’s scores were ~ .04 lower). Support vector machines trained on the features generated by the SA lexicons reached macro F1-scores of .62 and .58 for the two tasks. These computationally-generated annotations also offered the possibility to explore the LOBO corpus from a “distant reading” perspective. Here, SA methods provided more detailed results, e.g. mapping basic emotions to literary genres (see Figure 1). Such fine-grained visualizations were not possible with ML methods. However, they proved more efficient in distinguishing the ratings of the reviews (see Table 1).