Summary

This chapter was all about Transformer, a powerful neural network architecture designed for sequence-to-sequence tasks. Its key ingredient, self-attention, lets the model focus on the most important parts of the information it’s looking at in a sequence.

We worked on two NLP projects: sentiment analysis and text generation using two state-of-the-art Transformer models, BERT and GPT. We observed an elevated performance compared to what we did in the last chapter. We also learned how to fine-tune these Transformers with the Hugging Face library, a one-stop shop for loading pre-trained models, performing different NLP tasks, and fine-tuning models on your own data. Plus, it throws in some bonus tools for chopping up text, checking how well the model did, and even generating some text of its own.

In the next chapter, we will focus on another OpenAI cutting-edge model, CLIP, and will implement natural language-based image search.