This PhD will be developed within the framework of the 3-year DeepKnowledge project, with the aim of investigating and developing enabling techniques, methods and novel deep learning tools towards Natural Language Understanding and Generation. The project will pay special attention to Transformer-based pre-trained language models to address the lack of training data and knowledge resources for many NLP tasks and domains, focusing mostly on Spanish, English, Basque, Catalan and Galician. Additionally, this subproject will aim to develop new models for Basque, Catalan, Galician and Spanish to address NLU tasks that are also capable of dealing with text generation tasks, as well as models that are trained in a multi-task fashion, which have shown to generalize better and yield good results in zero-shot and few-shot scenarios.