The original skip-gram algorithm

The skip-gram algorithm discussed up to this point in the book is actually an improvement over the original skip-gram algorithm proposed in the original paper by Mikolov and others, published in 2013. In this paper, the algorithm did not use an intermediate hidden layer to learn the representations. In contrast, the original algorithm used two different embedding or projection layers (the input and output embeddings in Figure 4.1) and defined a cost function derived from the embeddings themselves:

Figure 4.1: The original skip-gram algorithm without hidden layers

The original negative sampled loss was defined as follows:

Here, v is the input embeddings layer, v' is the output word embeddings layer, corresponds to the embedding vector for the word w_i in the input embeddings layer and corresponds to the word vector for the word w_i in the output embeddings layer.

is the noise distribution, from which we sample noise samples (for example, it can be as...