Customer churn is the loss of clients or customers (also known as customer attrition, customer turnover, or customer defection). This concept was initially used within the telecommunications industry when many subscribers switched to other service providers. However, it has become a very important issue in other areas of business, such as banks, internet service providers, insurance companies, and so on. Well, two of the primary reasons for churn are customer dissatisfaction and cheaper and/or better offers from the competition.

As you can see in Figure 1, there are four possible contracts with the customer in a business industry: contractual, non-contractual, voluntary, and involuntary. The full cost of customer churn includes both the lost revenue and the (tele-) marketing costs involved with replacing those customers...

In ML, we observe an algorithm's performance in two stages: learning and inference. The ultimate target of the learning stage is to prepare and describe the available data, also called the feature vector, which is used to train the model.

The learning stage is one of the most important stages, but it is also truly time-consuming. It involves preparing a list of vectors, also called feature vectors (vectors of numbers representing the value of each feature), from the training data after transformation so that we can feed them to the learning algorithms. On the other hand, training data also sometimes contains impure information that needs some pre-processing, such as cleaning.

Once we have the feature vectors, the next step in this stage is preparing (or writing/reusing) the learning algorithm. The next important step is training...

LR is one of the most widely used classifiers to predict a binary response. It is a linear ML method, as described in Chapter 1, Analyzing Insurance Severity Claim. The loss function is the formulation given by the logistic loss:

For the LR model, the loss function is the logistic loss. For a binary classification problem, the algorithm outputs a binary LR model such that, for a given new data point, denoted by x, the model makes predictions by applying the logistic function:

In the preceding equation, z = W^TX and if f(W^TX)>0.5, the outcome is positive; otherwise, it is negative.

Note that compared to linear regression, logistic regression provides you with a higher classification accuracy. Moreover, it is a flexible way to regularize a model for custom adjustment...

SVM is also used widely for large-scale classification (that is, binary as well as multinomial) tasks. Besides, it is also a linear ML method, as described in Chapter 1, Analyzing Insurance Severity Claim. The linear SVM algorithm outputs an SVM model, where the loss function used by SVM can be defined using the hinge loss, as follows:

The linear SVMs in Spark are trained with an L2 regularization, by default. However, it also supports L1 regularization, by which the problem itself becomes a linear program.

Now, suppose we have a set of new data points x; the model makes predictions based on the value of w^Tx. By default, if w^Tx≥0, then the outcome is positive, and negative otherwise.

Now that we already know the SVMs working principle, let's start using the Spark-based implementation of SVM. Let's start...

DTs are commonly considered a supervised learning technique used for solving classification and regression tasks.

More technically, each branch in a DT represents a possible decision, occurrence, or reaction, in terms of statistical probability. Compared to naive Bayes, DTs are a far more robust classification technique. The reason is that at first, the DT splits the features into training and test sets. Then, it produces a good generalization to infer the predicted labels or classes. Most interestingly, a DT algorithm can handle both binary and multiclass classification problems.

For instance, in the following example figure, DTs learn from the admission data to approximate a sine curve with a set of if...else decision rules. The dataset contains the record of each student who applied for admission, say, to an American university. Each record contains...

As described in Chapter 1, Analyzing Insurance Severity Claim, Random Forest is an ensemble technique that takes a subset of observations and a subset of variables to build decision trees—that is, an ensemble of DTs. More technically, it builds several decision trees and integrates them together to get a more accurate and stable prediction.

This is a direct consequence, since by maximum voting from a panel of independent juries, we get the final prediction better than the best jury (see the preceding figure). Now that we already know the working principle of RF, let's start using the Spark-based implementation of RF. Let's start by importing the required packages and libraries:

import org.apache.spark._
import org.apache.spark.sql.SparkSession
import org.apache...

From the preceding results, it can be seen that LR and SVM models have the same but higher false positive rate compared to Random Forest and DT. So we can say that DT and Random Forest have better accuracy overall in terms of true positive counts. Let's see the validity of the preceding statement with prediction distributions on pie charts for each model:

Now, it's worth mentioning that using random forest, we are actually getting high accuracy, but it's a very resource, as well as time-consuming job; the training, especially, takes a considerably longer time as compared to LR and SVM.

Therefore, if you don't have higher memory or computing power, it is recommended to increase the Java heap space prior to running this code to avoid OOM errors.

Finally, if you want to deploy the best model (that is, Random Forest in our...