Pre-trained models

The most recent approach to NLU is based on the idea that much of the information required to understand natural language can be made available to many different applications by processing generic text (such as internet text) to create a baseline model for the language. Some of these models are very large and are based on tremendous amounts of data. To apply these models to a specific application, the generic model is adapted to the application through the use of application-specific training data, through a process called fine-tuning. Because the baseline model already contains a vast amount of general information about the language, the amount of training data can be considerably less than the training data required for some of the traditional approaches. These popular technologies include BERT and its many variations and Generative Pre-trained Transformers (GPTs) and their variations.

Pre-trained models will be discussed in detail in Chapter 11.