Word embedding ‒ origins and fundamentals

Wikipedia defines word embedding as the collective name for a set of language modeling and feature learning techniques in natural language processing (NLP) where words or phrases from a vocabulary are mapped to vectors of real numbers.

Deep learning models, like other machine learning models, typically don't work directly with text; the text needs to be converted to numbers instead. The process of converting text to numbers is a process called vectorization. An early technique for vectorizing words was one-hot encoding, which you have learned about in Chapter 1, Neural Network Foundations with TensorFlow 2.0. As you will recall, a major problem with one-hot encoding is that it treats each word as completely independent from all the others, since similarity between any two words (measured by the dot product of the two-word vectors) is always zero.

The dot product is an algebraic operation that operates on two vectors...