Machine learning models are mathematical systems that share many common features. Even if, sometimes, they have been defined only from a theoretical viewpoint, research advancement allows us to apply several concepts to better understand the behavior of complex systems such as deep neural networks. In this chapter, we're going to introduce and discuss some fundamental elements that some skilled readers may already know, but that, at the same time, offer several possible interpretations and applications.
In particular, in this chapter we're discussing the main elements of:
- Data-generating processes
- Finite datasets
- Training and test split strategies
- Cross-validation
- Capacity, bias, and variance of a model
- Vapnik-Chervonenkis theory
- Cramér-Rao bound
- Underfitting and overfitting
- Loss and cost functions
- Regularization
