Regularization using a word2vec embedding
In this recipe, we will use a pre-trained word2vec embedding to improve the results of a task thanks to transfer learning. We will compare the results to the initial Training a GRU recipe from Chapter 8, on the IMDb dataset for review classification.
Getting ready
A word2vec is a rather old type of word embedding in the NLP landscape and has been widely used in many NLP tasks. While recent techniques are sometimes more powerful, the word2vec approach remains efficient and cost-effective.
Without getting into the details of word2vec, a commonly used model is a 300-dimensional embedding; each word in the vocabulary is embedded into a vector of 300 values.
word2vec is usually trained on a large corpus of texts. There are two main approaches for training a word2vec that can be roughly described as follows:
- Continuous bag of words (CBOW): Uses the context of surrounding words in a sentence to predict a missing word
- skip-gram...
