In both supervised and unsupervised learning problems, there will always be a dataset, defined as a finite set of real vectors with m features each:

Considering that our approach is always probabilistic, we need to assume each X as drawn from a statistical multivariate distribution, D, that is commonly known as a data generating process (the probability density function is often denoted as p_data(x)). For our purposes, it's also useful to add a very important condition upon the whole dataset X: we expect all samples to be independent and identically distributed (i.i.d). This means that all variables belong to the same distribution, D, and considering an arbitrary subset of k values, it happens that the following is true:

It's fundamental to understand that all machine learning tasks are based on the assumption of working with well-defined distributions...

A parametric model can be split into two parts: a static structure and a dynamic set of parameters. The former is determined by the choice of a specific algorithm and is normally immutable (except in the cases when the model provides some remodeling functionalities), while the latter is the objective of our optimization. Considering n unbounded parameters, they generate an n-dimensional space (imposing bounds results in a subspace without relevant changes in our discussion) where each point, together with the immutable part of the estimator function, represents a learning hypothesis H (associated with a specific set of parameters):

The goal of a parametric learning process is to find the best hypothesis whose corresponding prediction error is at minimum and the residual generalization ability is enough to avoid overfitting. In the following diagram, we can see an...

Imagine that you need to design a spam-filtering algorithm, starting from this initial (over-simplistic) classification based on two parameters:

We have collected 200 email messages (X) (for simplicity, we consider p₁ and p_{2 as} mutually exclusive) and we need to find a couple of probabilistic hypotheses (expressed in terms of p₁ and p₂), to determine the following:

We also assume the conditional independence of both terms (it means that h_p1 and h_p2 contribute in conjunction to spam in the same way as they would alone).

For example, we could think about rules (hypotheses) like so: "If there are more than five blacklisted words" or "If the message is less than 20 characters...

When working with the majority of machine learning algorithms (in particular, supervised ones), it's important to train the model with a dataset containing almost the same number of elements for each class. Remember that our goal is training models that can generalize in the best way for all of the possible classes and supposes that we have a binary dataset containing 1,000 samples with a proportion (0.95, 0.05). There are many scenarios where this proportion is very common. For example, a spam detector can collect lots of spam emails, but it's much more difficult to have access to personally accepted emails. However, we can suppose that some users (a very small percentage) decided to share anonymous regular messages so that our dataset consists of 5% non-spam entries.

Now, let's consider a static algorithm that always outputs the label 0 (for example...

A machine learning problem can also be analyzed in terms of information transfer or exchange. Our dataset is composed of n features, which are considered independent (for simplicity, even if it's often a realistic assumption) and drawn from n different statistical distributions. Therefore, there are n probability density functions p_i(x) which must be approximated through other n q_i(x) functions. In any machine learning task, it's very important to understand how two corresponding distributions diverge and what the amount of information we lose is when approximating the original dataset.

The most useful measure in information theory (as well as in machine learning) is called...

In this chapter, we have introduced some main concepts about machine learning. We started with some basic mathematical definitions so that we have a clear view of data formats, standards, and certain kinds of functions. This notation will be adopted in the rest of the chapters in this book, and it's also the most diffused in technical publications. We also discussed how scikit-learn seamlessly works with multi-class problems, and when a strategy is preferable to another.

The next step was the introduction of some fundamental theoretical concepts regarding learnability. The main questions we tried to answer were: how can we decide if a problem can be learned by an algorithm and what is the maximum precision we can achieve? PAC learning is a generic but powerful definition that can be adopted when defining the boundaries of an algorithm. A PAC learnable problem, in...