Solving for your model size

Now that you’ve picked your best base model(s), you understand its pretraining regime, and you identified your dataset and its overall size in the last chapter, let’s start to understand the sizes of models you can target!

You may remember that in Chapter 1, we introduced a core concept called the scaling laws. Introduced by Kaplan et al. in 2020, this bold idea suggests a formal relationship between the overall sizes of your compute training cluster, your dataset, and your model. Prior to Kaplan, most machine learning practitioners had understood there to be a general relationship between these three, but his team took the bold task of proving this empirically via power laws.

The basic thing you need to understand can be demonstrated with a simple graphic. To train your model well, both in terms of producing the highest accuracy you can and in getting the most value out of your overall compute budget, it’s helpful to think about...