Chapter 7: Supervised Learning: Predicting Customer Churn

Activity 13: Performing OSE from OSEMN

1. Import the necessary libraries.

   import pandas as pd
   import numpy as np
   import matplotlib.pyplot as plt
   import seaborn as sns

   Read the dataset using pandas read.csv. and look at the first few rows of the DataFrame:

   data= pd.read_csv(r'Telco_Churn_Data.csv')
   data.head(5)

2. Check the length and shape of the data.

   len(data)
   data.shape

   The length should be 4708 and the shape should be (4708, 15).

3. Check for any missing values present in the data set and use the info method to check missing values in each of the columns.

   data.isnull().values.any()

   This will return True, implying that missing values are present.

   data.info()

   This gives the following output:

   

   Figure 7.58: Output of data.info

4. Rename all the columns in a readable format. Convert all the columns names with a space to _, for example, rename Target Code to Target_Code.

   data.columns=data.columns.str.replace(' ','_')
   data.columns

5. Check the descriptive statistics of the data

   data.describe()

6. Check the descriptive statistics of Categorical variable

   data.describe(include='object')

7. Change the data type of Target_Code, Condition_of_Current_Handset, and Current_TechSupComplaints columns from continuous to categorical object type:

   data['Target_Code']=data.Target_Code.astype('object')
   data['Condition_of_Current_Handset']=data.Condition_of_Current_Handset.astype('object')
   data['Current_TechSupComplaints']=data.Current_TechSupComplaints.astype('object')
   data['Target_Code']=data.Target_Code.astype('int64')
   data.describe(include='object')

   This gives the following output:

   

   Figure 7.59: Output of describe function for categorical variables

8. Check the percentage of missing values and then impute the values of both Complaint_Code and Condition_of_Current_Handset with the most occurring values:

   round(data.isnull().sum()/len(data)*100,2)

   

   Figure 7.60: Checking percentage of missing values

   data.Complaint_Code.value_counts()

   

   Figure 7.61: Checking missing values in Complaint_Code

   data.Condition_of_Current_Handset.value_counts()

   

   Figure 7.62: Checking missing values in Condition_of_Current_Handset

   data['Complaint_Code']=data['Complaint_Code'].fillna(value='Billing Problem')
   data['Condition_of_Current_Handset']=data['Condition_of_Current_Handset'].fillna(value=1)
   data['Condition_of_Current_Handset']=data.Condition_of_Current_Handset.astype('object')

9. Perform data exploration by initially exploring the customer Target_Churn variable:

   data['Target_Churn'].value_counts(0)
   data['Target_Churn'].value_counts(1)*100
   summary_churn = data.groupby('Target_Churn')
   summary_churn.mean()

10. Find the correlation among different variables:

    corr = data.corr()
    plt.figure(figsize=(15,8))
    sns.heatmap(corr, 
                xticklabels=corr.columns.values,
                yticklabels=corr.columns.values,annot=True)
    corr

    From the plots, you will observe that Avg_Calls_Weekdays and Avg_Calls are highly correlated, which makes sense since they represent the same thing—average calls. Current_Bill_Amt seems to be correlated with both variables, which is as expected, since the more you talk the higher your bill will be.

11. Perform univariate and bivariate analysis.

    Here's the univariate analysis:

    f, axes = plt.subplots(ncols=3, figsize=(15, 6))
    sns.distplot(data.Avg_Calls_Weekdays, kde=True,  color="darkgreen", ax=axes[0]).set_title('Avg_Calls_Weekdays')
    axes[0].set_ylabel('No of Customers')
    sns.distplot(data.Avg_Calls, kde=True,color="darkblue", ax=axes[1]).set_title('Avg_Calls')
    axes[1].set_ylabel('No of Customers')
    sns.distplot(data.Current_Bill_Amt, kde=True, color="maroon", ax=axes[2]).set_title('Current_Bill_Amt')
    axes[2].set_ylabel('No of Customers')

    And here's the bivariate analysis:

    Code for the plot of Complaint_Code versus Target_Churn, is given here:

    plt.figure(figsize=(17,10))
    p=sns.countplot(y="Complaint_Code", hue='Target_Churn', data=data,palette="Set2")
    legend = p.get_legend()
    legend_txt = legend.texts
    legend_txt[0].set_text("No Churn")
    legend_txt[1].set_text("Churn")
    p.set_title('Customer Complaint Code Distribution')

    From this plot, you'll observe that call quality and billing problems are the two main reasons for customer churn.

    Cod for the plot of Acct_Plan_Subtype versus Target_Churn is given here:

    plt.figure(figsize=(15,4))
    p=sns.countplot(y="Acct_Plan_Subtype", hue='Target_Churn', data=data,palette="Set2")
    legend = p.get_legend()
    legend_txt = legend.texts
    legend_txt[0].set_text("No Churn")
    legend_txt[1].set_text("Churn")
    p.set_title('Customer Acct_Plan_Subtype Distribution')

    Code for the plot of Current_TechSupComplaints versus Target_Churn is given here:

    plt.figure(figsize=(15,4))
    p=sns.countplot(y="Current_TechSupComplaints", hue='Target_Churn', data=data,palette="Set2")
    legend = p.get_legend()
    legend_txt = legend.texts
    legend_txt[0].set_text("No Churn")
    legend_txt[1].set_text("Churn")
    p.set_title('Customer Current_TechSupComplaints Distribution')

    Code for the plot of Avg_Days_Delinquent versus Target_Code is given here.

    plt.figure(figsize=(15,4))
    ax=sns.kdeplot(data.loc[(data['Target_Code'] == 0),'Avg_Days_Delinquent'] , color=sns.color_palette("Set2")[0],shade=True,label='no churn')
    ax=sns.kdeplot(data.loc[(data['Target_Code'] == 1),'Avg_Days_Delinquent'] , color=sns.color_palette("Set2")[1],shade=True, label='churn')
    ax.set(xlabel='Average No of Days Deliquent/Defaluted from paying', ylabel='Frequency')
    plt.title('Average No of Days Deliquent/Defaluted from paying - churn vs no churn')

    From this plot, you'll observe that if the average number of days delinquent is more than 16 days, customers start to churn.

    Code for the plot of Account_Age versus Target_Code is given here:

    plt.figure(figsize=(15,4))
    ax=sns.kdeplot(data.loc[(data['Target_Code'] == 0),'Account_Age'] , color=sns.color_palette("Set2")[0],shade=True,label='no churn')
    ax=sns.kdeplot(data.loc[(data['Target_Code'] == 1),'Account_Age'] , color=sns.color_palette("Set2")[1],shade=True, label='churn')
    ax.set(xlabel='Account_Age', ylabel='Frequency')
    plt.title('Account_Age - churn vs no churn')

    From this plot, you'll observe that during the initial 15-20 days of opening an account, the amount of customer churn increases; however, after 20 days, the churn rate declines.

    Code for the plot of Percent_Increase_MOM vs Target_Code is given here:

    plt.figure(figsize=(15,4))
    ax=sns.kdeplot(data.loc[(data['Target_Code'] == 0),'Percent_Increase_MOM'] , color=sns.color_palette("Set2")[0],shade=True,label='no churn')
    ax=sns.kdeplot(data.loc[(data['Target_Code'] == 1),'Percent_Increase_MOM'] , color=sns.color_palette("Set2")[1],shade=True, label='churn')
    ax.set(xlabel='Percent_Increase_MOM', ylabel='Frequency')
    plt.title('Percent_Increase_MOM- churn vs no churn')

    From this plot, you will note that customers who have Percent_Increase_MOM within a range of −ve% to +ve% have a greater likelihood of churning.

Activity 14: Performing MN of OSEMN

1. Import the RandomForestClassifier, train_test_split, and numpy library:

   from sklearn.ensemble import RandomForestClassifier
   from sklearn.model_selection import train_test_split
   import numpy as np

2. Encode the columns:

   data["Acct_Plan_Subtype"] = data["Acct_Plan_Subtype"].astype('category').cat.codes
   data["Complaint_Code"] = data["Complaint_Code"].astype('ca
   tegory').cat.codes
   data[["Acct_Plan_Subtype","Complaint_Code"]].head()

3. Split the data into a training and testing set:

   target = 'Target_Code'
   X = data.drop(['Target_Code','Target_Churn'], axis=1)
   y=data[target]
   X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.15, random_state=123, stratify=y)

4. Perform feature selection using the random forest classifier:

   forest=RandomForestClassifier(n_estimators=500,random_state=1)
   forest.fit(X_train,y_train)
   importances=forest.feature_importances_
   features = data.drop(['Target_Code','Target_Churn'],axis=1).columns
   indices = np.argsort(importances)[::-1]
   plt.figure(figsize=(15,4))
   plt.title("Feature importances using Random Forest")
   plt.bar(range(X_train.shape[1]), importances[indices],
          color="r",  align="center")
   plt.xticks(range(X_train.shape[1]), features[indices], rotation='vertical',fontsize=15)
   plt.xlim([-1, X_train.shape[1]])
   plt.show()

5. Import statsmodels:

   import statsmodels.api as sm
   top7_features = ['Avg_Days_Delinquent','Percent_Increase_MOM','Avg_Calls_Weekdays','Current_Bill_Amt','Avg_Calls','Complaint_Code','Account_Age']
   logReg = sm.Logit(y_train, X_train[top7_features])
   logistic_regression = logReg.fit()

6. Find out the parameters:

   logistic_regression.summary
   logistic_regression.params

7. Create a function to compute the cost function:

   coef = logistic_regression.params
   def y (coef, Avg_Days_Delinquent,Percent_Increase_MOM,Avg_Calls_Weekdays,Current_Bill_Amt,Avg_Calls,Complaint_Code,Account_Age) : 
       final_coef=coef[0]*Avg_Days_Delinquent+ coef[1]*Percent_Increase_MOM+coef[2]*Avg_Calls_Weekdays+coef[3]*Current_Bill_Amt+ coef[4]*Avg_Calls+coef[5]*Complaint_Code+coef[6]*Account_Age
       return final_coef

8. Input the given attributes of the customer to the function to obtain the output:

   Avg_Days_Delinquent:40, Percent_Increase_MOM:5, Avg_Calls_Weekdays:39000, Current_Bill_Amt:12000, Avg_Calls:9000, Complaint_Code:0, Account_Age:17
   y1 = y(coef, 40, 5, 39000,12000,9000,0,17)
   p = np.exp(y1) / (1+np.exp(y1))
   p