IU Incubator
IMA-XR

Interactive Motion Agent for XR

Project Description

IMA-XR focuses on research and development in the context of multimodal generative artificial intelligence (AI) methods for interactive AI agents in virtual (VR) and extended realities (XR).

While many current agent systems are purely text-based, and the creation of 3D agents for XR demands manual work and expert knowledge in animation and scripting, IMA-XR aims at facilitating the creation of AI agents via multimodal generative AI methods without necessitating extensive expertise, manual work or very high computational resources. In terms of (long-term) target audience, this could be interesting for e.g. developers and content creators for XR and robotics applications, educational institutions and training providers, especially in areas involving movement demonstration for training purposes, and rehabilitation and sports trainers, for motor learning.

We build on existing work in context-specific motion understanding and description (of tracked real motion) and motion generation, whereas context is currently provided by language and spatial conditions and focus on research of interactive and compute-resource efficient approaches to enhance usability and adaptability to different application scenarios.

This research aims to work towards customizable approaches to bridge the gap between conversational (language-based) agents and the spatial (three-dimensional, motion-based) interaction that is crucial for immersive XR experiences including digital characters. While we consider virtual agents in this project, this research is also a pre-step in the direction of intuitive real-world interactions with embodied agents (such as robots).

Research questions:

How can semantically encoded motion and language be combined to enable natural, continuous conversations with an AI agent in interactive XR environments, while maintaining low-latency responsiveness and user-centric adaptation?
How can spatial / environmental encoding and vocabulary expansion be developed to enhance context-aware motion-language interactions across diverse XR use cases, potentially leveraging multimodal inputs (e.g., gaze, audio)?
How can current multimodal agent systems for context-specific motion understanding, description and generation be customized with minimal computational resources, while balancing technical efficiency with user experience requirements (e.g., motion naturalness, comfort)?

Duration of the Project

10/2025 – 09/2027

Additional Information

Bertram Taetz
Professor of Data Science & AI
He completed his Ph.D. in mathematics and physics at Ruhr-Universität Bochum in 2012. From 2013 to 2025 he worked at DFKI leading a team for human motion capture, modeling, and analysis based on AI methods. From 2015 to 2020 he was deputy head, and from 2021 to 2022 head of the wearHEALTH group at RPTU. In 2019, he co-founded the sci-track GmbH. His areas of competence are machine learning (probabilistic methods, representation learning, continual learning), computer vision and sensor fusion, human-machine interaction and human motion capture and analysis for applications in health, medicine, robotics, AR/VR/XR.

Gabriele-Bleser Taetz
Professor of Robotics
From 2008 to 2024 she worked as a senior researcher, from 2013 to 2014 as deputy head, in the Augmented Vision Department at DFKI. Before, she worked as a researcher at Fraunhofer IGD in Darmstadt. From 2014 to 2021, she led the wearHEALTH group at RPTU. In 2019, she co-founded sci-track GmbH, where she is managing director. Her research areas are computer vision, sensor fusion, modeling, capturing and analysis of human motion, for applications in human-machine interaction, biomechanics, ergonomics, health, XR and robotics.

Janki Dodiya
Professor of Augmented/Virtual Reality and Human-Computer Interaction
She has over a decade of experience designing immersive technologies for Aerospace, Transportation, and Education. Previously, she was a Senior Researcher at DLR (2012–2021), working on teleoperation applications for on-orbit servicing and interaction design for autonomous vehicles. Her research interests are XR, HCI, AI for XR, avatar representation, usability, and societal impact of emerging technologies.

Armin Grasnick
Professor of Augmented & Virtual Reality
After studying technical optics, he began his professional career as a research assistant for the design of high-performance lenses. During New Economy, he became self-employed with a start-up for the development of 3D displays and subsequently founded other companies with a 3D/VR focus in Germany and abroad. In the course of his career, he has registered several patents and written a few books on virtual reality.