IU Incubator IMA-XR

IMA-XR focuses on research and development in the context of multimodal generative artificial intelligence (AI) methods for interactive AI agents in virtual (VR) and extended realities (XR).

While many current agent systems are purely text-based, and the creation of 3D agents for XR demands manual work and expert knowledge in animation and scripting, IMA-XR aims at facilitating the creation of AI agents via multimodal generative AI methods without necessitating extensive expertise, manual work or very high computational resources. In terms of (long-term) target audience, this could be interesting for e.g. developers and content creators for XR and robotics applications, educational institutions and training providers, especially in areas involving movement demonstration for training purposes, and rehabilitation and sports trainers, for motor learning.

We build on existing work in context-specific motion understanding and description (of tracked real motion) and motion generation, whereas context is currently provided by language and spatial conditions and focus on research of interactive and compute-resource efficient approaches to enhance usability and adaptability to different application scenarios.

This research aims to work towards customizable approaches to bridge the gap between conversational (language-based) agents and the spatial (three-dimensional, motion-based) interaction that is crucial for immersive XR experiences including digital characters. While we consider virtual agents in this project, this research is also a pre-step in the direction of intuitive real-world interactions with embodied agents (such as robots).

Research questions:

• How can semantically encoded motion and language be combined to enable natural, continuous conversations with an AI agent in interactive XR environments, while maintaining low-latency responsiveness and user-centric adaptation?

• How can spatial / environmental encoding and vocabulary expansion be developed to enhance context-aware motion-language interactions across diverse XR use cases, potentially leveraging multimodal inputs (e.g., gaze, audio)?

• How can current multimodal agent systems for context-specific motion understanding, description and generation be customized with minimal computational resources, while balancing technical efficiency with user experience requirements (e.g., motion naturalness, comfort)?