Examining pre-training and fine-tuning processes

Pre-training and fine-tuning are two key stages when training LLMs such as GPT-3.5. Pre-training is like a student’s general education in that it covers a broad range of subjects to provide foundational knowledge. Fine-tuning, on the other hand, is like a student later specializing in a specific subject in college, refining their skills for a particular field. In the context of LLMs, pre-training sets the broad base, and fine-tuning narrows the focus to excel in specific tasks. In this section, we’ll look at pre-training and fine-tuning to see how fine-tuning adds value:

Figure 3.2 – Two-step LLM training process

Let’s provide an overview of the two stages.

Pre-training process

Pre-training is the initial phase of training a language model. During this phase, the model learns from a massive amount of text data, often referred to as the “pre-training corpus.”...