Evaluating foundation models
As we’ve discussed many times in this book so far, the primary reason to engage in large-scale training is that open source models aren’t cutting it for you. Before you start your own large-scale training project, you should have already completed the following steps:
- Tested an open source model on your specific use case
- Identified performance gaps
- Fine-tuned that same open source model on a small subset of your data
- Identified smaller performance gaps
The point is that you should have some empirical reason to believe that the open source model solves some of your business problem but not all of it. You need to also empirically prove that small-scale fine-tuning is in the same boat; it should increase system performance but still leave room for improvement. This entire next section is about evaluating that room for improvement. Let’s try to understand how we can evaluate foundation models.
As you are no...
