In this chapter, we will use the Python libraries Matplotlib, pandas, NumPy, scikit-learn, and Feature-engine. If you need to install Python, the free Anaconda Python distribution (https://www.anaconda.com/) includes most numerical computing libraries.

feature-engine can be installed with pip as follows:

pip install feature-engine

If you use Anaconda, you can install feature-engine with conda:

conda install -c conda-forge feature_engine

Note

The recipes from this chapter were created using the latest versions of the Python libraries at the time of publishing. You can check the versions in the requirements.txt file in the accompanying GitHub repository, at https://github.com/PacktPublishing/Python-Feature-engineering-Cookbook-Third-Edition/blob/main/requirements.txt.

We will use the Credit Approval dataset from the UCI Machine Learning Repository (https://archive.ics.uci.edu/), licensed under the CC BY 4.0 creative commons attribution: https://creativecommons.org/licenses/by/4.0/legalcode. You’ll find the dataset at this link: http://archive.ics.uci.edu/dataset/27/credit+approval.

I downloaded and modified the data as shown in this notebook: https://github.com/PacktPublishing/Python-Feature-engineering-Cookbook-Third-Edition/blob/main/ch01-missing-data-imputation/credit-approval-dataset.ipynb

We will also use the air passenger dataset located in Facebook’s Prophet GitHub repository (https://github.com/facebook/prophet/blob/main/examples/example_air_passengers.csv), licensed under the MIT license: https://github.com/facebook/prophet/blob/main/LICENSE

I modified the data as shown in this notebook: https://github.com/PacktPublishing/Python-Feature-engineering-Cookbook-Third-Edition/blob/main/ch01-missing-data-imputation/air-passengers-dataset.ipynb

You’ll find a copy of the modified data sets in the accompanying GitHub repository: https://github.com/PacktPublishing/Python-Feature-engineering-Cookbook-Third-Edition/blob/main/ch01-missing-data-imputation/

Complete Case Analysis (CCA), also called list-wise deletion of cases, consists of discarding observations with missing data. CCA can be applied to both categorical and numerical variables. With CCA, we preserve the distribution of the variables after the imputation, provided the data is missing at random and only in a small proportion of observations. However, if data is missing across many variables, CCA may lead to the removal of a large portion of the dataset.

Note

Use CCA only when a small number of observations are missing and you have good reasons to believe that they are not important to your model.

How to do it...

Let’s begin by making some imports and loading the dataset:

1. Let’s import pandas, matplotlib, and the train/test split function from scikit-learn:

   import matplotlib.pyplot as plt
   import pandas as pd
   from sklearn.model_selection import train_test_split

2. Let’s load and display the dataset described in the Technical requirements section:

   data = pd.read_csv("credit_approval_uci.csv")
   data.head()

   In the following image, we see the first 5 rows of data:

Figure 1.1 – First 5 rows of the dataset

3. Let’s proceed as we normally would if we were preparing the data to train machine learning models; by splitting the data into a training and a test set:

   X_train, X_test, y_train, y_test = train_test_split(
       data.drop("target", axis=1),
       data["target"],
       test_size=0.30,
       random_state=42,
   )

4. Let’s now make a bar plot with the proportion of missing data per variable in the training and test sets:

   fig, axes = plt.subplots(
       2, 1, figsize=(15, 10), squeeze=False)
   X_train.isnull().mean().plot(
       kind='bar', color='grey', ax=axes[0, 0], title="train")
   X_test.isnull().mean().plot(
       kind='bar', color='black', ax=axes[1, 0], title="test")
   axes[0, 0].set_ylabel('Fraction of NAN')
   axes[1, 0].set_ylabel('Fraction of NAN')
   plt.show()

   The previous code block returns the following bar plots with the fraction of missing data per variable in the training (top) and test sets (bottom):

Figure 1.2 – Proportion of missing data per variable

5. Now, we’ll remove observations if they have missing values in any variable:

   train_cca = X_train.dropna()
   test_cca = X_test.dropna()

Note

pandas’ dropna()drops observations with any missing value by default. We can remove observations with missing data in a subset of variables like this: data.dropna(subset=["A3", "A4"]).

6. Let’s print and compare the size of the original and complete case datasets:

   print(f"Total observations: {len(X_train)}")
   print(f"Observations without NAN: {len(train_cca)}")

   We removed more than 200 observations with missing data from the training set, as shown in the following output:

   Total observations: 483
   Observations without NAN: 264

7. After removing observations from the training and test sets, we need to align the target variables:

   y_train_cca = y_train.loc[train_cca.index]
   y_test_cca = y_test.loc[test_cca.index]

   Now, the datasets and target variables contain the rows without missing data.

8. To drop observations with missing data utilizing feature-engine, let’s import the required transformer:

   from feature_engine.imputation import DropMissingData

9. Let’s set up the imputer to automatically find the variables with missing data:

   cca = DropMissingData(variables=None, missing_only=True)

10. Let’s fit the transformer so that it finds the variables with missing data:

    cca.fit(X_train)

11. Let’s inspect the variables with NAN that the transformer found:

    cca.variables_

    The previous command returns the names of the variables with missing data:

    ['A1', 'A2', 'A3', 'A4', 'A5', 'A6', 'A7', 'A8', 'A9', 'A10', 'A14']

12. Let’s remove the rows with missing data in the training and test sets:

    train_cca = cca.transform(X_train)
    test_cca = cca.transform(X_test)

    Use train_cca.isnull().sum() to corroborate the absence of missing data in the complete case dataset.

13. DropMissingData can automatically adjust the target after removing missing data from the training set:

    train_c, y_train_c = cca.transform_x_y( X_train, y_train)
    test_c, y_test_c = cca.transform_x_y(X_test, y_test)

The previous code removed rows with nan from the training and test sets and then re-aligned the target variables.

Note

To remove observations with missing data in a subset of variables, use DropMissingData(variables=['A3', 'A4']). To remove rows with nan in at least 5% of the variables, use DropMissingData(threshold=0.95).

How it works...

In this recipe, we plotted the proportion of missing data in each variable and then removed all observations with missing values.

We used pandas isnull() and mean() methods to determine the proportion of missing observations in each variable. The isnull() method created a Boolean vector per variable with True and False values indicating whether a value was missing. The mean() method took the average of these values and returned the proportion of missing data.

We used pandas plot.bar() to create a bar plot of the fraction of missing data per variable. In Figure 1.2, we saw the fraction of nan per variable in the training and test sets.

To remove observations with missing values in any variable, we used pandas’ dropna(), thereby obtaining a complete case dataset.

Finally, we removed missing data using Feature-engine’s DropMissingData(). This imputer automatically identified and stored the variables with missing data from the train set when we called the fit() method. With the transform() method, the imputer removed observations with nan in those variables. With transform_x_y(), the imputer removed rows with nan from the data sets and then realigned the target variable.

See also

If you want to use DropMissingData() within a pipeline together with other Feature-engine or scikit-learn transformers, check out Feature-engine’s Pipeline: https://Feature-engine.trainindata.com/en/latest/user_guide/pipeline/Pipeline.html. This pipeline can align the target with the training and test sets after removing rows.

Mean or median imputation consists of replacing missing data with the variable’s mean or median value. To avoid data leakage, we determine the mean or median using the train set, and then use these values to impute the train and test sets, and all future data.

Scikit-learn and Feature-engine learn the mean or median from the train set and store these parameters for future use out of the box.

In this recipe, we will perform mean and median imputation using pandas, scikit-learn, and feature-engine.

Note

Use mean imputation if variables are normally distributed and median imputation otherwise. Mean and median imputation may distort the variable distribution if there is a high percentage of missing data.

How to do it...

Let’s begin this recipe:

1. First, we’ll import pandas and the required functions and classes from scikit-learn and feature-engine:

   import pandas as pd
   from sklearn.model_selection import train_test_split
   from sklearn.impute import SimpleImputer
   from sklearn.compose import ColumnTransformer
   from feature_engine.imputation import MeanMedianImputer

2. Let’s load the dataset that we prepared in the Technical requirements section:

   data = pd.read_csv("credit_approval_uci.csv")

3. Let’s split the data into train and test sets with their respective targets:

   X_train, X_test, y_train, y_test = train_test_split(
       data.drop("target", axis=1),
       data["target"],
       test_size=0.3,
       random_state=0,
   )

4. Let’s make a list with the numerical variables by excluding variables of type object:

   numeric_vars = X_train.select_dtypes(
       exclude="O").columns.to_list()

   If you execute numeric_vars, you will see the names of the numerical variables: ['A2', 'A3', 'A8', 'A11', 'A14', 'A15'].

5. Let’s capture the variables’ median values in a dictionary:

   median_values = X_train[
       numeric_vars].median().to_dict()

Tip

Note how we calculate the median using the train set. We will use these values to replace missing data in the train and test sets. To calculate the mean, use pandas mean() instead of median().

If you execute median_values, you will see a dictionary with the median value per variable: {'A2': 28.835, 'A3': 2.75, 'A8': 1.0, 'A11': 0.0, 'A14': 160.0, 'A15': 6.0}.

6. Let’s replace missing data with the median:

   X_train_t = X_train.fillna(value=median_values)
   X_test_t = X_test.fillna(value=median_values)

   If you execute X_train_t[numeric_vars].isnull().sum() after the imputation, the number of missing values in the numerical variables should be 0.

Note

pandas fillna() returns a new dataset with imputed values by default. To replace missing data in the original DataFrame, set the inplace parameter to True: X_train.fillna(value=median_values, inplace=True).

Now, let’s impute missing values with the median using scikit-learn.

7. Let’s set up the imputer to replace missing data with the median:

   imputer = SimpleImputer(strategy="median")

Note

To perform mean imputation, set SimpleImputer() as follows: imputer = SimpleImputer(strategy = "mean").

8. We restrict the imputation to the numerical variables by using ColumnTransformer():

   ct = ColumnTransformer(
       [("imputer", imputer, numeric_vars)],
       remainder="passthrough",
       force_int_remainder_cols=False,
   ).set_output(transform="pandas")

Note

Scikit-learn can return numpy arrays, pandas DataFrames, or polar frames, depending on how we set out the transform output. By default, it returns numpy arrays.

9. Let’s fit the imputer to the train set so that it learns the median values:

   ct.fit(X_train)

10. Let’s check out the learned median values:

    ct.named_transformers_.imputer.statistics_

    The previous command returns the median values per variable:

    array([ 28.835,   2.75,   1.,   0., 160.,   6.])

11. Let’s replace missing values with the median:

    X_train_t = ct.transform(X_train)
    X_test_t = ct.transform(X_test)

12. Let’s display the resulting training set:

    print(X_train_t.head())

    We see the resulting DataFrame in the following image:

Figure 1.3 – Training set after the imputation. The imputed variables are marked by the imputer prefix; the untransformed variables show the prefix remainder

Finally, let’s perform median imputation using feature-engine.

13. Let’s set up the imputer to replace missing data in numerical variables with the median:

    imputer = MeanMedianImputer(
        imputation_method="median",
        variables=numeric_vars,
    )

Note

To perform mean imputation, change imputation_method to "mean". By default MeanMedianImputer() will impute all numerical variables in the DataFrame, ignoring categorical variables. Use the variables argument to restrict the imputation to a subset of numerical variables.

14. Fit the imputer so that it learns the median values:

    imputer.fit(X_train)

15. Inspect the learned medians:

    imputer.imputer_dict_

    The previous command returns the median values in a dictionary:

    {'A2': 28.835, 'A3': 2.75, 'A8': 1.0, 'A11': 0.0, 'A14': 160.0, 'A15': 6.0}

16. Finally, let’s replace the missing values with the median:

    X_train = imputer.transform(X_train)
    X_test = imputer.transform(X_test)

Feature-engine’s MeanMedianImputer() returns a DataFrame. You can check that the imputed variables do not contain missing values using X_train[numeric_vars].isnull().mean().

How it works...

In this recipe, we replaced missing data with the variable’s median values using pandas, scikit-learn, and feature-engine.

We divided the dataset into train and test sets using scikit-learn’s train_test_split() function. The function takes the predictor variables, the target, the fraction of observations to retain in the test set, and a random_state value for reproducibility, as arguments. It returned a train set with 70% of the original observations and a test set with 30% of the original observations. The 70:30 split was done at random.

To impute missing data with pandas, in step 5, we created a dictionary with the numerical variable names as keys and their medians as values. The median values were learned from the training set to avoid data leakage. To replace missing data, we applied pandas’ fillna() to train and test sets, passing the dictionary with the median values per variable as a parameter.

To replace the missing values with the median using scikit-learn, we used SimpleImputer() with the strategy set to "median". To restrict the imputation to numerical variables, we used ColumnTransformer(). With the remainder argument set to passthrough, we made ColumnTransformer() return all the variables seen in the training set in the transformed output; the imputed ones followed by those that were not transformed.

Note

ColumnTransformer() changes the names of the variables in the output. The transformed variables show the prefix imputer and the unchanged variables show the prefix remainder.

In step 8, we set the output of the column transformer to pandas to obtain a DataFrame as a result. By default, ColumnTransformer() returns numpy arrays.

Note

From version 1.4.0, scikit-learn transformers can return numpy arrays, pandas DataFrames, or polar frames as a result of the transform() method.

With fit(), SimpleImputer() learned the median of each numerical variable in the train set and stored them in its statistics_ attribute. With transform(), it replaced the missing values with the medians.

To replace missing values with the median using Feature-engine, we used the MeanMedianImputer() with the imputation_method set to median. To restrict the imputation to a subset of variables, we passed the variable names in a list to the variables parameter. With fit(), the transformer learned and stored the median values per variable in a dictionary in its imputer_dict_ attribute. With transform(), it replaced the missing values, returning a pandas DataFrame.

We typically impute categorical variables with the most frequent category, or with a specific string. To avoid data leakage, we find the frequent categories from the train set. Then, we use these values to impute the train, test, and future datasets. scikit-learn and feature-engine find and store the frequent categories for the imputation, out of the box.

In this recipe, we will replace missing data in categorical variables with the most frequent category, or with an arbitrary string.

How to do it...

To begin, let’s make a few imports and prepare the data:

1. Let’s import pandas and the required functions and classes from scikit-learn and feature-engine:

   import pandas as pd
   from sklearn.model_selection import train_test_split
   from sklearn.impute import SimpleImputer
   from sklearn.compose import ColumnTransformer
   from feature_engine.imputation import CategoricalImputer

2. Let’s load the dataset that we prepared in the Technical requirements section:

   data = pd.read_csv("credit_approval_uci.csv")

3. Let’s split the data into train and test sets and their respective targets:

   X_train, X_test, y_train, y_test = train_test_split(
       data.drop("target", axis=1),
       data["target"],
       test_size=0.3,
       random_state=0,
   )

4. Let’s capture the categorical variables in a list:

   categorical_vars = X_train.select_dtypes(
       include="O").columns.to_list()

5. Let’s store the variables’ most frequent categories in a dictionary:

   frequent_values = X_train[
       categorical_vars].mode().iloc[0].to_dict()

6. Let’s replace missing values with the frequent categories:

   X_train_t = X_train.fillna(value=frequent_values)
   X_test_t = X_test.fillna(value=frequent_values)

Note

fillna() returns a new DataFrame with the imputed values by default. We can replace missing data in the original DataFrame by executing X_train.fillna(value=frequent_values, inplace=True).

7. To replace missing data with a specific string, let’s create an imputation dictionary with the categorical variable names as the keys and an arbitrary string as the values:

   imputation_dict = {var:
        "no_data" for var in categorical_vars}

   Now, we can use this dictionary and the code in step 6 to replace missing data.

Note

With pandas value_counts() we can see the string added by the imputation. Try executing, for example, X_train["A1"].value_counts().

Now, let’s impute missing values with the most frequent category using scikit-learn.

8. Let’s set up the imputer to find the most frequent category per variable:

   imputer = SimpleImputer(strategy='most_frequent')

Note

SimpleImputer() will learn the mode for numerical and categorical variables alike. But in practice, mode imputation is done for categorical variables only.

9. Let’s restrict the imputation to the categorical variables:

   ct = ColumnTransformer(
       [("imputer",imputer, categorical_vars)],
       remainder="passthrough"
       ).set_output(transform="pandas")

Note

To impute missing data with a string instead of the most frequent category, set SimpleImputer() as follows: imputer = SimpleImputer(strategy="constant", fill_value="missing").

10. Fit the imputer to the train set so that it learns the most frequent values:

    ct.fit(X_train)

11. Let’s take a look at the most frequent values learned by the imputer:

    ct.named_transformers_.imputer.statistics_

    The previous command returns the most frequent values per variable:

    array(['b', 'u', 'g', 'c', 'v', 't', 'f', 'f', 'g'], dtype=object)

12. Finally, let’s replace missing values with the frequent categories:

    X_train_t = ct.transform(X_train)
    X_test_t = ct.transform(X_test)

    Make sure to inspect the resulting DataFrames by executing X_train_t.head().

Note

The ColumnTransformer() changes the names of the variables. The imputed variables show the prefix imputer and the untransformed variables the prefix remainder.

Finally, let’s impute missing values using feature-engine.

13. Let’s set up the imputer to replace the missing data in categorical variables with their most frequent value:

    imputer = CategoricalImputer(
        imputation_method="frequent",
        variables=categorical_vars,
    )

Note

With the variables parameter set to None, CategoricalImputer() will automatically impute all categorical variables found in the train set. Use this parameter to restrict the imputation to a subset of categorical variables, as shown in step 13.

14. Fit the imputer to the train set so that it learns the most frequent categories:

    imputer.fit(X_train)

Note

To impute categorical variables with a specific string, set imputation_method to missing and fill_value to the desired string.

15. Let’s check out the learned categories:

    imputer.imputer_dict_

    We can see the dictionary with the most frequent values in the following output:

    {'A1': 'b',
     'A4': 'u',
     'A5': 'g',
     'A6': 'c',
     'A7': 'v',
     'A9': 't',
     'A10': 'f',
     'A12': 'f',
     'A13': 'g'}

16. Finally, let’s replace the missing values with frequent categories:

    X_train_t = imputer.transform(X_train)
    X_test_t = imputer.transform(X_test)

    If you want to impute numerical variables with a string or the most frequent value using CategoricalImputer(), set the ignore_format parameter to True.

CategoricalImputer() returns a pandas DataFrame as a result.

How it works...

In this recipe, we replaced missing values in categorical variables with the most frequent categories or an arbitrary string. We used pandas, scikit-learn, and feature-engine.

In step 5, we created a dictionary with the variable names as keys and the frequent categories as values. To capture the frequent categories, we used pandas mode(), and to return a dictionary, we used pandas to_dict(). To replace the missing data, we used pandas fillna(), passing the dictionary with the variables and their frequent categories as parameters. There can be more than one mode in a variable, that’s why we made sure to capture only one of those values by using .iloc[0].

To replace the missing values using scikit-learn, we used SimpleImputer() with the strategy set to most_frequent. To restrict the imputation to categorical variables, we used ColumnTransformer(). With remainder set to passthrough, we made ColumnTransformer() return all the variables present in the training set as a result of the transform() method .

Note

ColumnTransformer() changes the names of the variables in the output. The transformed variables show the prefix imputer and the unchanged variables show the prefix remainder.

With fit(), SimpleImputer() learned the variables’ most frequent categories and stored them in its statistics_ attribute. With transform(), it replaced the missing data with the learned parameters.

SimpleImputer() and ColumnTransformer() return NumPy arrays by default. We can change this behavior with the set_output() parameter.

To replace missing values with feature-engine, we used the CategoricalImputer() with imputation_method set to frequent. With fit(), the transformer learned and stored the most frequent categories in a dictionary in its imputer_dict_ attribute. With transform(), it replaced the missing values with the learned parameters.

Unlike SimpleImputer(), CategoricalImputer() will only impute categorical variables, unless specifically told not to do so by setting the ignore_format parameter to True. In addition, with feature-engine transformers we can restrict the transformations to a subset of variables through the transformer itself. For scikit-learn transformers, we need the additional ColumnTransformer() class to apply the transformation to a subset of the variables.

We can replace missing data with an arbitrary value. Commonly used values are 999, 9999, or -1 for positive distributions. This method is used for numerical variables. For categorical variables, the equivalent method is to replace missing data with an arbitrary string, as described in the Imputing categorical variables recipe.

When replacing missing values with arbitrary numbers, we need to be careful not to select a value close to the mean, the median, or any other common value of the distribution.

Note

We’d use arbitrary number imputation when data is not missing at random, use non-linear models, or when the percentage of missing data is high. This imputation technique distorts the original variable distribution.

In this recipe, we will impute missing data with arbitrary numbers using pandas, scikit-learn, and feature-engine.

How to do it...

Let’s begin by importing the necessary tools and loading the data:

1. Import pandas and the required functions and classes:

   import pandas as pd
   from sklearn.model_selection import train_test_split
   from sklearn.impute import SimpleImputer
   from feature_engine.imputation import ArbitraryNumberImputer

2. Let’s load the dataset described in the Technical requirements section:

   data = pd.read_csv("credit_approval_uci.csv")

3. Let’s separate the data into train and test sets:

   X_train, X_test, y_train, y_test = train_test_split(
       data.drop("target", axis=1),
       data["target"],
       test_size=0.3,
       random_state=0,
   )

   We will select arbitrary values greater than the maximum value of the distribution.

4. Let’s find the maximum value of four numerical variables:

   X_train[['A2','A3', 'A8', 'A11']].max()

   The previous command returns the following output:

   A2     76.750
   A3     26.335
   A8     28.500
   A11    67.000
   dtype: float64

   We’ll use 99 for the imputation because it is bigger than the maximum values of the numerical variables in step 4.

5. Let’s make a copy of the original DataFrames:

   X_train_t = X_train.copy()
   X_test_t = X_test.copy()

6. Now, we replace the missing values with 99:

   X_train_t[["A2", "A3", "A8", "A11"]] = X_train_t[[
       "A2", "A3", "A8", "A11"]].fillna(99)
   X_test_t[["A2", "A3", "A8", "A11"]] = X_test_t[[
       "A2", "A3", "A8", "A11"]].fillna(99)

Note

To impute different variables with different values using pandas fillna(), use a dictionary like this: imputation_dict = {"A2": -1, "A3": -1, "A8": 999, "A11": 9999}.

Now, we’ll impute missing values with an arbitrary number using scikit-learn.

1. Let’s set up imputer to replace missing values with 99:

   imputer = SimpleImputer(strategy='constant', fill_value=99)

Note

If your dataset contains categorical variables, SimpleImputer() will add 99 to those variables as well if any values are missing.

2. Let’s fit imputer to a slice of the train set containing the variables to impute:

   vars = ["A2", "A3", "A8", "A11"]
   imputer.fit(X_train[vars])

3. Replace the missing values with 99 in the desired variables:

   X_train_t[vars] = imputer.transform(X_train[vars])
   X_test_t[vars] = imputer.transform(X_test[vars])

   Go ahead and check the lack of missing values by executing X_test_t[["A2", "A3", "A8", "A11"]].isnull().sum().

   To finish, let’s impute missing values using feature-engine.

4. Let’s set up the imputer to replace missing values with 99 in 4 specific variables:

   imputer = ArbitraryNumberImputer(
       arbitrary_number=99,
       variables=["A2", "A3", "A8", "A11"],
   )

Note

ArbitraryNumberImputer() will automatically select all numerical variables in the train set for imputation if we set the variables parameter to None.

5. Finally, let’s replace the missing values with 99:

   X_train = imputer.fit_transform(X_train)
   X_test = imputer.transform(X_test)

Note

To impute different variables with different numbers, set up ArbitraryNumberImputer() as follows: ArbitraryNumberImputer(imputater_dict = {"A2": -1, "A3": -1, "A8": 999, "A11": 9999}).

We have now replaced missing data with arbitrary numbers using three different open-source libraries.

How it works...

In this recipe, we replaced missing values in numerical variables with an arbitrary number using pandas, scikit-learn, and feature-engine.

To determine which arbitrary value to use, we inspected the maximum values of four numerical variables using pandas’ max(). We chose 99 because it was greater than the maximum values of the selected variables. In step 5, we used pandas fillna() to replace the missing data.

To replace missing values using scikit-learn, we utilized SimpleImputer(), with the strategy set to constant, and specified 99 in the fill_value argument. Next, we fitted the imputer to a slice of the train set with the numerical variables to impute. Finally, we replaced missing values using transform().

To replace missing values with feature-engine we used ArbitraryValueImputer(), specifying the value 99 and the variables to impute as parameters. Next, we applied the fit_transform() method to replace missing data in the train set and the transform() method to replace missing data in the test set.