Classical approaches to learning word representation

In this section, we will discuss some of the classical approaches used for numerically representing words. It is important to have an understanding of the alternatives to word vectors, as these methods are still used in the real world, especially when limited data is available.

More specifically, we will discuss common representations, such as one-hot encoding and Term Frequency-Inverse Document Frequency (TF-IDF).

One-hot encoded representation

One of the simpler ways of representing words is to use the one-hot encoded representation. This means that if we have a vocabulary of size V, for each i^th word w_i, we will represent the word w_i with a V-length vector [0, 0, 0, …, 0, 1, 0, …, 0, 0, 0] where the i^th element is 1 and other elements are 0. As an example, consider this sentence:

Bob and Mary are good friends.

The one-hot encoded representation of each word might look like this:

Bob: [1...