A typical semi-supervised scenario is not very different from a supervised one. Let's suppose we have a data generating process, p_data:

However, contrary to a supervised approach, we have only a limited number N of samples drawn from p_data and provided with a label, as follows:

Instead, we have a larger amount (M) of unlabeled samples drawn from the marginal distribution p(x):

In general, there are no restrictions on the values of N and M; however, a semi-supervised problem arises when the number of unlabeled samples is much higher than the number of complete samples. If we can draw N >> M labeled samples from p_data, it's probably useless to keep on working with semi-supervised approaches and preferring classical supervised methods is likely to be the best choice. The extra complexity we need is justified by M >> N, which is a...

Generative Gaussian mixtures is an inductive algorithm for semi-supervised clustering. Let's suppose we have a labeled dataset (X_l, Y_l) containing N samples (drawn from p_data) and an unlabeled dataset X_u containing M >> N samples (drawn from the marginal distribution p(x)). It's not necessary that M >> N, but we want to create a real semi-supervised scenario, with only a few labeled samples. Moreover, we are assuming that all unlabeled samples are consistent with p_data. This can seem like a vicious cycle, but without this assumption, the procedure does not have a strong mathematical foundation. Our goal is to determine a complete p(x|y) distribution using a generative model. In general, it's possible to use different priors, but we are now employing multivariate Gaussians to model our data:

Thus, our model parameters...

As explained at the beginning of this chapter, in many real life problems, it's cheaper to retrieve unlabeled samples, rather than correctly labeled ones. For this reason, many researchers worked to find out the best strategies to carry out a semi-supervised classification that could outperform the supervised counterpart. The idea is to train a classifier with a few labeled samples and then improve its accuracy after adding weighted unlabeled samples. One of the best results is the Contrastive Pessimistic Likelihood Estimation (CPLE) algorithm, proposed by M. Loog (in Loog M., Contrastive Pessimistic Likelihood Estimation for Semi-Supervised Classification, arXiv:1503.00269).

Before explaining this algorithm, an introduction is necessary. If we have a labeled dataset (X, Y) containing N samples, it's possible to define...

When we discussed the cluster assumption, we also defined the low-density regions as boundaries and the corresponding problem as low-density separation. A common supervised classifier which is based on this concept is a Support Vector Machine (SVM), the objective of which is to maximize the distance between the dense regions where the samples must be. For a complete description of linear and kernel-based SVMs, please refer to Bonaccorso G., Machine Learning Algorithms, Packt Publishing; however, it's useful to remind yourself of the basic model for a linear SVM with slack variables ξ_i:

This model is based on the assumptions that y_i can be either -1 or 1. The slack variables ξ_i or soft-margins are variables, one for each sample, introduced to reduce the strength imposed by the original condition...

Another approach to the same problem is offered by the TSVM, proposed by T. Joachims (in Transductive Inference for Text Classification using Support Vector Machines, Joachims T., ICML Vol. 99/1999). The idea is to keep the original objective with two sets of slack variables: the first for the labeled samples and the other for the unlabeled ones:

As this is a transductive approach, we need to consider the unlabeled samples as variable-labeled ones (subject to the learning process), imposing a constraint similar to the supervised points. As for the previous algorithm, we assume we have N labeled samples and M unlabeled ones; therefore, the conditions become as follows:

The first constraint is the classical SVM one and it works only on labeled samples. The second one uses the variable y^(u)_j with the corresponding slack variables...

In this chapter, we introduced semi-supervised learning, starting from the scenario and the assumptions needed to justify the approaches. We discussed the importance of the smoothness assumption when working with both supervised and semi-supervised classifiers in order to guarantee a reasonable generalization ability. Then we introduced the clustering assumption, which is strictly related to the geometry of the datasets and allows coping with density estimation problems with a strong structural condition. Finally, we discussed the manifold assumption and its importance in order to avoid the curse of dimensionality.

The chapter continued by introducing a generative and inductive model: Generative Gaussian mixtures, which allow clustering labeled and unlabeled samples starting from the assumption that the prior probabilities are modeled by multivariate Gaussian distributions...