From Generative Models to Large Language Models
In this chapter, you will learn about Generative Language Models (GLMs) and Large Language Model (LLMs). After this, you will learn how to pre-train any language model, such as Generated Pre-trained Transformer 2 (GPT-2), on your own text and use it for tasks such as Natural Language Generation (NLG). You will learn the basics of Text-to-Text Transfer Transformer (T5) models, conduct a hands-on multitask learning experiment with T5, and train a Multilingual T5 (mT5) model on your own Machine Translation (MT) data. After finishing this chapter, you will have an overview of GLMs and their various use cases in text2text applications, such as summarization, paraphrasing, multitask learning, zero-shot learning, and MT.
The following topics will be covered in this chapter:
- Working with GLMs
- Working with text-to-text models
- AR language model training
- GLM training
- NLG using AR models
