PUB - Publikationen an der Universität Bielefeld

Language processing does not happen in isolation, but is often embedded in a rich non-linguistic visual and social context. Yet, although many psycholinguistic studies have investigated the close interplay between language and the visual context, the role of social aspects and listener characteristics in real-time language processing remains largely elusive. The present thesis aims at closing this gap.
Taking extant literature regarding the incrementality of language processing, the close interplay between visual and linguistic context and the relevance for and effect of social aspects on language comprehension into account, we argue for the necessity to extend investigations on the influence of social information and listener characteristics on real-time language processing. Crucially, we moreover argue for the inclusion of social information and listener characteristics into real-time language processing accounts. Up-to-date, extant accounts on language comprehension remain elusive about the influence of social cues and listener characteristics on real-time language processing. Yet a more comprehensive approach that takes these aspects into account is highly desirable given that psycholinguistics aims at describing how language processing happens in real-time in the mind of the comprehender. In 6 eye-tracking studies, this thesis hence investigated the effect of two distinct visual contextual cues on real-time language processing and thematic role assignment in emotionally valenced non-canonical German sentences. We are using emotional facial expressions of a speaker as a visual social cue and depicted actions as a visual contextual cue that is directly mediated by the linguistic input. Crucially, we are also investigating the effect of the age of the listener as one type of listener characteristics in testing children and older and younger adults.
In our studies, participants were primed with a positive emotional facial expression (vs. a non-emotional / negative expression). Following this they inspected a target scene depicting two potential agents either performing or not performing an action towards a patient. This scene was accompanied by a related positively valenced German Object-Verb-Adverb-Subject sentence (e.g.,: The ladybug(accusative object, patient) tickles happily the cat(nominative object, agent).). Anticipatory eye-movements to the agent of the action, i.e., the sentential subject in sentence end position (vs. distractor agent), were measured in order to investigate if, to what extent and how rapidly positive emotional facial expressions and depicted actions can facilitate thematic role assignment in children and older and younger adults. Moreover, given the complex nature of emotional facial expressions, we also investigated if the naturalness of the emotional face has an influence on the integration of this social cue into real-time sentence processing. We hence used a schematic depiction of an emotional face, i.e., a happy smiley, in half of the studies and a natural human emotional face in the remaining studies.
Our results showed that all age groups could reliably use the depicted actions as a cue to facilitate sentence processing and to assign thematic roles even before the target agent had been mentioned. Crucially, only our adult listener groups could also use the emotional facial expression for real-time sentence processing. When the natural human facial expression instead of the schematic smiley was used to portray the positive emotion, the use of the social cue was even stronger. Nevertheless, our results have also suggested that the depicted action is a stronger cue than the social cue, i.e., the emotional facial expression, for both adult age groups. Children on the other hand do not yet seem to be able to also use emotional facial expressions as visual social cues for language comprehension. Interestingly, we also found time course differences regarding the integration of the two cues into real-time sentence comprehension. Compared to younger adults, both older adults and children were delayed by one word region in their visual cue effects.
Our on-line data is further supported by accuracy results. All age groups answered comprehension questions for ‘who is doing what to whom’ more accurately when an action was depicted (vs. was not depicted). However, only younger adults made use of the emotional cue for answering the comprehension questions, although to a lesser extent than they used depicted actions.
In conclusion, our findings suggest for the first time that different non-linguistic cues, i.e., more direct referential cues such as depicted actions and more indirect social cues such as emotional facial expressions, are integrated into situated language processing to different degrees. Crucially, the time course and strength of the integration of these cues varies as a function of age.
Hence our findings support our argument regarding the inclusion of social cues and listener characteristics into real-time language processing accounts. Based on our own results we have therefore outlined at the end of this thesis, how an account of real-time language comprehension that already takes the influence of visual context such as depicted actions into account (but fails to include social aspects and listener characteristics) can be enriched to also include the effects of emotional facial expressions and listener characteristics such as age.