PUB - Publikationen an der Universität Bielefeld

Human communication is inherently multimodal, with gestures playing a crucial role in conveying meaning beyond words alone. However, current gesture generation systems predominantly focus on visual naturalness while neglecting the communicative pur- pose of gestures, resulting in movements that appear fluid yet lack semantic informa- tion. This thesis addresses the fundamental challenge of creating non-verbal behaviors that are not only visually natural but also semantically meaningful and contextually grounded, thereby enhancing multimodal communication. It establishes a new cate- gory of gesture generation approaches called context-driven, that combines the strengths of both intent-driven and speech-driven gesture generation methods. Through five inter- connected papers, this work develops and evaluates several novel frameworks. First, the AQ-GT model demonstrates how quantization and hybrid GRU-Transformer archi- tectures can generate highly realistic beat gestures, while the AQ-GT-A model extends this work by incorporating form and meaning annotations to guide the gesture genera- tion process. An evaluation study of both models highlights the weaknesses of current speech-driven gesture generation and shifts the focus of this thesis from speech-driven to context-driven approaches. Based on this, the TF-JAX-IK algorithm provides a real- time inverse kinematics solution for mapping high-level gesture concepts onto natural human motion. This TF-JAX-IK algorithm is then used for the ImaGGen framework, which introduces a Semantic Planning approach that generates contextually grounded semantic gestures by analyzing visual input without requiring extensive training data. The evaluation of ImaGGens contextually grounded gestures shows that they signifi- cantly improve the delivery of information, particularly when speech is ambiguous, while maintaining naturalness through the integration of speech-driven beat gestures. The work presented here makes several theoretical and practical contributions to the field of multimodal interaction and gesture generation. It challenges the prevailing as- sumption of recent years that visual naturalness alone is sufficient for effective gesture generation, emphasizing instead the crucial role of communicative intent and semantic grounding in producing meaningful gestures. The developed methods have direct ap- plications in virtual reality and assistive communication tools, enabling virtual agents with ImaGGen-like capabilities to deliver clearer explanations and support more natu- ral interactions. Finally, this thesis focuses on an open-source research approach, with a transparent and extendable code basis for all presented papers.