PUB - Publikationen an der Universität Bielefeld

In this paper, we argue that empirical research on genuine linguistic topics, such as on the production of multimodal utterances in the speaker and the interpretation of the multimodal signals in the interlocutor, can greatly benefit from the use of virtual reality technologies. Established methodologies for research on multimodal interactions, such as the presentation of pre-recorded 2D videos of interaction partners as stimuli and the recording of interaction partners using multiple 2D video cameras have crucial shortcomings regarding ecological validity and the precision of measurements that can be achieved. In addition, these methodologies enforce restrictions on the researcher. The stimuli, for example, are not very interactive and thus not as close to natural interactions as ultimately desired. Also, the analysis of 2D video recordings requires intensive manual annotations, often frame-by-frame, which negatively affects the feasible number of interactions which can be included in a study. The technologies bundled under the term virtual reality offer exciting possibilities for the linguistic researcher: gestures can be tracked without being restricted to fixed perspectives, annotation can be done on large corpora (semi-)automatically and virtual characters can be used to produce specific linguistic stimuli in a repetitive but interactive fashion. Moreover, immersive 3D visualizations can be used to recreate a simulation of the recorded interactions by fusing the raw data with theoretic models to support an iterative data-driven development of linguistic theories. This paper discusses the potential of virtual reality technologies for linguistic research and provides examples for the application of the methodology.