The kernel trick
To learn how the kernel trick allows us to do feature construction implicitly and efficiently, we will first have to learn what a kernel is.
What is a kernel?
The simplest way to think about a kernel is to consider it as a mapping that takes two vectors as input and returns a scalar. It is a mapping that maps 
. This means that a kernel is a function 
, with the input vectors being 
and 
The value of 
is a real number. This means that the inner product 
is an example of a kernel function.
That is a high-level mathematical definition of what a kernel is, but what is the intuition behind this? An 
kernel function applied to the vectors 
and 
is typically used to measure the similarity between those vectors. Consequently, we usually want our kernel function to have its largest values when 
and 
are most similar and its lowest values when and are least similar. We want the function to decrease smoothly and monotonically in between those two scenarios.
