The candidate will participate in the LUMINOUS project, which aims to contribute towards the creation of the next generation of Language Augmented Extended Reality (XR) systems and applications. LUMINOUS is a 3-year coordinated project funded by the European Commission (HORIZON-CL4-2023-HUMAN-01-CNECT). The project’s main goal is to create an integrated language-augmented multimodal platform that adapts to undefined individual user needs and unseen environments.  Exploitation of multimodal LLMs will be central for an efficient generalisation over new situations and objects. 

More specifically the goal of this position is to make use of augmented MLLMs to generate user-directed instructions in a grounded multimodal environment by incorporating common-sense reasoning and situation-specific knowledge into XR systems and further adapting them to specific domains and environments. The candidate will focus on, among other things:

• Adaptation of models to specific domains using fine-tuning  techniques
• Adaptation of models using prompt-based techniques, such as instruction tuning
• Grounding of MLLMs into explicit knowledge
• Improving MLLMs generation of user-directed instructions by incorporating common sense reasoning and grounding

Although we have a well-defined set of goals for the project, there will also be ample room for the candidate to explore their own research interests and collaborate with other members of the HiTZ center.

The supervisors will be Oier Lopez de Lacalle and Jeremy Barnes. If you have any questions, please do not hesitate to contact at this address: recruitment.hitz@ehu.eus. Please include the job ID when contacting us.