The Continuous Bag-of-Words algorithm

The CBOW model has a working similar to the skip-gram algorithm with one significant change in the problem formulation. In the skip-gram model, we predicted the context words from the target word. However, in the CBOW model, we will predict the target from contextual words. Let's compare what data looks like for skip-gram and CBOW by taking the previous example sentence:

The dog barked at the mailman.

For skip-gram, data tuples—(input word, output word)—might look like this:

(dog, the), (dog, barked), (barked, dog), and so on.

For CBOW, data tuples would look like the following:

([the, barked], dog), ([dog, at], barked), and so on.

Consequently, the input of the CBOW has a dimensionality of 2 × m × D, where m is the context window size and D is the dimensionality of the embeddings. The conceptual model of CBOW is shown in Figure 3.13:

Figure 3.13: The CBOW model

We will not go into great details about the intricacies of CBOW...