In this chapter, we will utilize common libraries such as numpy, scikit-learn, and imbalanced-learn. The code and notebooks for this chapter are available on GitHub at https://github.com/PacktPublishing/Machine-Learning-for-Imbalanced-Data/tree/master/chapter02. You can just fire up the GitHub notebook using Google Colab by clicking on the Open in Colab icon at the top of this chapter’s notebook or by launching it from https://colab.research.google.com using the GitHub URL of the notebook.

Sampling involves selecting a subset of observations from a larger set of observations. In this chapter, we’ll initially focus on binary classification problems with two classes: the positive class and the negative class. The minority class has significantly fewer instances than the majority class. Later in this chapter, we will explore multi-class classification problems. Toward the end of this chapter, we will look into oversampling for multi-class classification problems.

Oversampling is a data balancing technique that generates more samples of the minority class. However, this can be easily scaled to work for any number of classes where there are multiple classes with an imbalance. Figure 2.1 shows how samples of minority and majority classes are imbalanced (a) initially and balanced (b) after applying an oversampling technique:

Figure 2.1 – An increase in the number of minority class samples after oversampling

The simplest strategy to balance the imbalance in a dataset is to randomly choose samples of the minority class and repeat or duplicate them. This is also called random oversampling with replacement.

To increase the number of minority class observations, we can replicate the minority class data observations enough times to balance the two classes. Does this sound too trivial? Yes, but it works. By increasing the number of minority class samples, random oversampling reduces the bias toward the majority class. This helps the model learn the patterns and characteristics of the minority class more effectively.

We will use random oversampling from the imbalanced-learn library. The fit_resample API from the RandomOverSampler class resamples the original dataset and balances it. The sampling_strategy parameter is used to specify the new ratio of various classes. For example, we could say sampling_strategy=1.0 to have an equal number of the two classes.

There are...

The main problem with random oversampling is that it duplicates the observations from the minority class. This can often cause overfitting. Synthetic Minority Oversampling Technique (SMOTE) [2] solves this problem of duplication by using a technique called interpolation.

Interpolation involves creating new data points in the range of known data points. Think of interpolation as being similar to the process of reproduction in biology. In reproduction, two individuals come together to produce a new individual with traits of both of them. Similarly, in interpolation, we pick two observations from the dataset and create a new observation by choosing a random point on the line joining the two selected points.

We oversample the minority class by interpolating synthetic examples. That prevents the duplication of minority samples while generating new synthetic observations similar to the known points. Figure 2.5 depicts how SMOTE works:

Figure 2.5 –...

Now, let’s look at some of the SMOTE variants, such as Borderline-SMOTE, SMOTE-NC, and SMOTEN. These variants apply the SMOTE algorithm to samples of a certain kind and may not always be applicable.

Borderline-SMOTE

Borderline-SMOTE [4] is a variation of SMOTE that generates synthetic samples from the minority class samples that are near the classification boundary, which divides the majority class from the minority class.

Why consider samples on the classification boundary?

The idea is that the examples near the classification boundary are more prone to misclassification than those far away from the decision boundary. Producing more such minority samples along the boundary would help the model learn better about the minority class. Intuitively, it is also true that the points away from the classification boundary likely won’t make the model a better classifier.

Here’s a step-by-step algorithm for Borderline-SMOTE:

1. We run a...

While SMOTE doesn’t distinguish between the density distribution of minority class samples, Adaptive Synthetic Sampling (ADASYN) [6] focuses on harder-to-classify minority class samples since they are in a low-density area. ADASYN uses a weighted distribution of the minority class based on the difficulty of classifying the observations. This way, more synthetic data is generated from harder samples:

Figure 2.11 – Illustration of how ADASYN works

Here, we can see the following:

• a) The majority and minority class samples are plotted
• b) Synthetic samples are generated depending on the hardness factor (explained later)

While SMOTE uses all samples from the minority class for oversampling uniformly, in ADASYN, the observations that are harder to classify are used more often.

Another difference between the two techniques is that, unlike SMOTE, ADASYN also uses the majority class observations while training KNN. It then...

Let’s examine how some popular models perform with the different oversampling techniques we’ve discussed. We’ll use two datasets for this comparison: one synthetic and one real-world dataset. We’ll evaluate the performance of four oversampling techniques, as well as no sampling, using logistic regression and random forest models.

You can find all the related code in this book’s GitHub repository. In Figure 2.15 and Figure 2.16, we can see the average precision score values for both models on the two datasets:

Figure 2.15 – Performance comparison of various oversampling techniques on a synthetic dataset

Figure 2.16 – Performance comparison of various oversampling techniques on the thyroid_sick dataset

Based on these plots, we can draw some useful conclusions:

• Effectiveness of oversampling: In general, using...

Now, let’s review some guidelines on how to navigate through the various oversampling techniques we went over and how these techniques differ from each other:

1. Train a model without applying any sampling techniques. This will be our model with baseline performance. Any oversampling technique we apply is expected to give a boost to this performance.
2. Start with random oversampling and add some shrinkage too. We may have to play with some values of shrinkage to see if the model’s performance improves.
3. When we have categorical features, we have a couple of options:
   1. Convert all categorical features into numerical features first using one-hot encoding, label encoding, feature hashing, or other feature transformation techniques.
   2. (Only for nominal categorical features) Use SMOTENC and SMOTEN directly on the data.
4. Apply various oversampling techniques – random oversampling, SMOTE, Borderline-SMOTE, and...

In multi-class classification problems, we have more than two classes or labels to be predicted, and hence more than one class may be imbalanced. This adds some more complexity to the problem. However, we can apply the same techniques to multi-class classification problems as well. The imbalanced-learn library provides the option to deal with multi-class classification in almost all the supported methods. We can choose from various sampling strategies using the sampling_strategy parameter. For multi-class classification, we can pass some fixed string values (called built-in strategies) to the sampling_strategy parameter in the SMOTE API. We can also pass a dictionary with the following:

• Keys as the class labels
• Values as the number of samples of that class

Here are the built-in strategies for sampling_strategy when using the parameter as a string:

• The minority strategy resamples only the minority class.
• The not...

In this chapter, we went through various oversampling techniques for dealing with imbalanced datasets and applied them using Python’s imbalanced-learn library (also called imblearn). We also saw the internal workings of some of the techniques by implementing them from scratch. While random oversampling generates new minority class samples by duplicating them, SMOTE-based techniques work by choosing random samples in the direction of nearest neighbors of the minority class samples. Though oversampling can potentially overfit the model on your data, it usually has more pros than cons, depending on the data and model.

We applied them to some of the synthesized and publicly available datasets and benchmarked their performance and effectiveness. We saw how different oversampling techniques may lead to model performance on a varying scale, so it becomes crucial to try a few different oversampling techniques to decide on the one that’s most optimal for our data.

1. Explore the two variants of SMOTE, namely KMeans-SMOTE and SVM-SMOTE, from the imbalanced-learn library, not discussed in this chapter. Compare their performance with vanilla SMOTE, Borderline-SMOTE, and ADASYN using the logistic regression and random forest models.
2. For a classification problem with two classes, let’s say the minority class to majority class ratio is 1:20. How should we balance this dataset? Should we apply the balancing technique at test or evaluation time? Please provide a reason for your answer.
3. Let’s say we are trying to build a model that can estimate whether a person can be granted a bank loan or not. Out of the 5,000 observations we have, only 500 people got the loan approved. To balance the dataset, we duplicate the approved people data and then split it into train, test, and validation datasets. Are there any issues with using this approach?
4. Data normalization helps in dealing with data imbalance. Is this true? Why...

1. Protecting Personal Data in Grab’s Imagery (2021), https://engineering.grab.com/protecting-personal-data-in-grabs-imagery.
2. N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, SMOTE: Synthetic Minority Over-sampling Technique, jair, vol. 16, pp. 321–357, Jun. 2002, doi: 10.1613/jair.953.
3. Live Site Incident escalation forecast (2023), https://medium.com/data-science-at-microsoft/live-site-incident-escalation-forecast-566763a2178.
4. H. Han, W.-Y. Wang, and B.-H. Mao, Borderline-SMOTE: A New Over-Sampling Method in Imbalanced Data Sets Learning, in Advances in Intelligent Computing, D.-S. Huang, X.-P. Zhang, and G.-B. Huang, Eds., in Lecture Notes in Computer Science, vol. 3644. Berlin, Heidelberg: Springer Berlin Heidelberg, 2005, pp. 878–887. doi: 10.1007/11538059_91.
5. P. Meiyappan and M. Bales, Position Paper: Reducing Amazon’s packaging waste using multimodal deep learning, (2021), article: https://www.amazon.science...