Discriminative and generative models

So far we have discussed logistic regression and a few extensions of it. In all cases, we tried to directly compute p( | ), that is, the probability of a given class knowing , which is some feature we measured to members of that class. In other words, we try to directly model the mapping from the independent variables to the dependent ones and then use a threshold to turn the (continuous) computed probability into a boundary that allows us to assign classes.

This approach is not unique. One alternative is to model first p( | ), that is, the distribution of for each class, and then assign the classes. This kind of model is called a generative classifier because we are creating a model from which we can generate samples from each class. On the contrary, logistic regression is a type of discriminative classifier since it tries to classify by discriminating classes but we cannot generate examples from each class.

We are not going to go into much detail...