When can we have an imbalance in datasets?
In this section, we’ll explore various situations and causes leading to an imbalance in datasets, such as rare event occurrences or skewed data collection processes:
- Inherent in the problem: Sometimes, the task we need to solve involves detecting outliers in datasets – for example, patients with a certain disease or fraud cases in a set of transactions. In such cases, the dataset is inherently imbalanced because the target events are rare to begin with.
- High cost of data collection while bootstrapping a machine learning solution: The cost of collecting data might be too high for certain classes. For example, collecting data on COVID-19 patients incurs high costs due to the need for specialized medical tests, protective equipment, and the ethical and logistical challenges of obtaining informed consent in a high-stress healthcare environment.
- Noisy labels for certain classes: This may happen when a lot of noise is introduced into the labels of the dataset for certain classes during data collection.
- Labeling errors: Errors in labeling can also contribute to data imbalance. For example, if some samples are mistakenly labeled as negative when they are positive, this can result in an imbalance in the dataset. Additionally, if a class is already inherently rare, human annotators might be biased and overlook the few examples of that rare class that do exist.
- Sampling bias: Data collection methods can sometimes introduce bias in the dataset. For example, if a survey is conducted in a specific geographical area or among a specific group of people, the resulting dataset may not be representative of the entire population.
- Data cleaning: During the data cleaning or filtering process, some classes or samples may be removed due to incomplete or missing data. This can result in an imbalance in the remaining dataset.
