Principal Component Analysis

Principal Component Analysis is a technique that takes datasets that have several correlated features and projects them onto a coordinate (axis) system that has fewer correlated features. These new, uncorrelated features (which I referred to before as a super-columns) are called principal components. The principal components serve as an alternative coordinate system to the original feature space that requires fewer features and captures as much variance as possible. If we refer back to our example with the cameras, the principal components are exemplified by the cameras themselves.

Put another way, the goal of the PCA is to identify patterns and latent structures within datasets in order to create new columns and use these columns instead of the original features. Just as in feature selection, if we start with a data matrix of size n x d where n is the number of observations and d is the number of original features, we are projecting this matrix onto a matrix...