LUMINOUS aims to contribute towards the creation of the next generation of Language Augmented XR systems and applications, where natural language-based communication and Large Language Models redefine the future interaction with novel extended reality (XR) technology and enhances understanding of the users’ situation and environment even in situations that are encountered for the first time.

The main objective is to create an integrated language-augmented multimodal platform that adapts to individual, not predefined user needs and unseen environments. This will enable future XR users to interact fluently with their environment while having instant access to constantly updated global as well as domain-specific knowledge sources to accomplish novel tasks. Such capabilities can have an unimaginable impact on future distance learning, training, entertainment, or provision of remote health services. In this direction, we aim to exploit Large Language Models (LLMs) for efficient generalisation over unseen situations and objects with effective means of communication supported by novel XR components. The LLM, in our vision, behaves like an oracle for describing novel tasks on user demand. These are then communicated through a speech interface and/or through an Avatar (e.g., Coach/Teacher) in terms of different visual aids and procedural steps for the accomplishment of the task. LLMs benefit in parallel in identifying new objects that were not part of their training data and then describing them in a way that they become visually recognizable. Through our cognitive environment, virtual avatars will facilitate
generalisable and situation-aware communication in multiple use cases and different sectors such as health (focus on rehabilitation and support of people with impairments), immersive learning, design reviewing with possible extensions to entertainment, and sport.
Organization:  European Commission
Main researcher: Didier Striker
Participants: 
DEUTSCHES FORSCHUNGSZENTRUM FUR KUNSTLICHE INTELLIGENZ GMBH - DFKI; LUDUS TECH SL; MINDESK SOCIETA A RESPONSABILITA LIMITATA; FRAUNHOFER GESELLSCHAFT ZUR FORDERUNG DER ANGEWANDTEN FORSCHUNG EV - Fraunhofer; FUNDACION CENTRO DE TECNOLOGIAS DE INTERACCION VISUAL Y COMUNICACIONES VICOMTECH - VICOM; UNIVERSITY COLLEGE DUBLIN, NATIONAL UNIVERSITY OF IRELAND, DUBLIN - NUID UCD; HYPERCLIQ IKE - HYPERCLIQ; RICOH INTERNATIONAL BV, Iñigo Alonso, Gorka Azkune, Jeremy Barnes, Ander Salaberria, Aitor Soroa