Dataset Preparation: Part One

In this chapter, we will begin to discuss what you’ll need in your dataset to start a meaningful pretraining project. This is the first of two parts on dataset preparation. It opens with some business guidance on finding a good use case for foundation modeling, where the data becomes instrumental. Then, focusing on the content of your dataset, we use qualitative and quantitative measures to compare it with datasets used to pretrain other top models. You’ll learn how to determine whether your datasets are “large enough” and “good enough” to boost accuracy while pretraining. We discuss bias identification and mitigation, along with multilingual and multimodal solutions.

In this chapter, we will cover the following topics:

• A business-level discussion on finding datasets and use cases for foundation modeling
• Evaluating your dataset by comparing it to ones available in the open source research community...