Building a transformer-based text generator with PyTorch
We built a transformer-based language model using PyTorch in the previous chapter. Because a language model models the probability of a certain word following a given sequence of words, we are more than halfway through building our own text generator. In this section, we will learn how to extend this language model as a deep generative model that can generate arbitrary yet meaningful sentences, given an initial textual cue in the form of a sequence of words.
Training the transformer-based language model
In the previous chapter, we trained a language model for 5 epochs. In this section, we will follow those exact same steps but will train the model for longer – 50 epochs. The goal here is to obtain a better-performing language model that can then generate realistic sentences. Please note that model training can take several hours. Hence, train it in the background; for example, overnight. In order to follow the...
