Research position on Augmented LLMs for eXtended Reality environments

The candidate will participate in the LUMINOUS project, which aims to contribute towards the creation of the next generation of Language Augmented Extended Reality (XR) systems and applications. LUMINOUS is a 3-year coordinated project funded by the European Commission (HORIZON-CL4-2023-HUMAN-01-CNECT). The project’s main goal is to create an integrated language-augmented multimodal platform that adapts to undefined individual user needs and unseen environments.  Exploitation of multimodal LLMs will be central for an efficient generalisation over new situations and objects. 

More specifically the goal of this position is to make use of augmented MLLMs to generate user-directed instructions in a grounded multimodal environment by incorporating common-sense reasoning and situation-specific knowledge into XR systems and further adapting them to specific domains and environments. The candidate will focus on, among other things:

  • Adaptation of models to specific domains using fine-tuning  techniques
  • Adaptation of models using prompt-based techniques, such as instruction tuning
  • Grounding of MLLMs into explicit knowledge
  • Improving MLLMs generation of user-directed instructions by incorporating common sense reasoning and grounding

Although we have a well-defined set of goals for the project, there will also be ample room for the candidate to explore their own research interests and collaborate with other members of the HiTZ center.

The supervisors will be Oier Lopez de Lacalle and Jeremy Barnes. If you have any questions, please do not hesitate to contact at this address: Please include the job ID when contacting us.

Job id: 

The candidate should preferably have a BSc degree in computer science, telecommunications engineering, mathematics or physics, and a MSc in language technologies and/or machine learning. We are looking for individuals who are passionate about natural language processing and have a strong background in computer science and related fields. Having a PhD in any of these areas will be highly valued. The applicants must demonstrate excellent communication skills in English.

Our ideal candidate has experience in machine learning, deep learning, and statistical analysis, as well as a strong proficiency in programming languages such as Python.

We welcome applicants from all backgrounds and are committed to creating an inclusive and supportive workplace.

Duration of Contract: 

3 years

Gross salary: 

37.900 euros per year.

Starting date: 
February 2024, with the choice of joining immediately.
Ez esleitua

To submit your application please follow this link.