Autoencoding Language Models

In the previous chapter, we looked at and studied how a typical transformer model can be used by Hugging Face’s Transformers. All of the topics in this book so far have included instructions on how to use pre-trained or pre-built models and less information has been given about specific models and their training.

In this chapter, we will learn how we can train autoencoding language models on any given language from scratch. This training will include pretraining and task-specific training of the models. First, we will start with learning about the BERT model and how it works. Then, we will train the language model using a simple and small corpus. Afterward, we will look at how the model can be used inside any Keras model.

For an overview of what will be learned in this chapter, we will cover the following topics:

• Bidirectional Encoder Representations from Transformers (BERT) – one of the autoencoding language models
• Autoencoding...