Summary

In this chapter, we introduced some of the core ideas that have dominated recent models for NLP, like the attention mechanism, contextual embeddings, and self-attention. We then used this foundation to learn about the transformer architecture and its internal components. We briefly discussed BERT and its family of architectures.

In the next section of the chapter, we presented a discussion on the transformer-based language models from OpenAI. We discussed the architectural and dataset-related choices for GPT and GPT-2. We also used the transformer package from Hugging Face to develop our own GPT-2-based text generation pipeline. We finally closed the chapter with a brief discussion on the latest and greatest language model, GPT-3. We discussed various motivations behind developing such a huge model and its long list of capabilities, which go beyond the list of traditionally tested benchmarks.

This chapter, along with Chapter 9, The Rise of Methods for Text Generation...