Visualizing the learned embeddings

Let us visualize the embedding in a 2D figure in order to get an understanding of how well they capture similarity and semantics. For that purpose, we need to reduce the number of dimension of the embedding, which is highly dimensional, to two dimensions without altering the structure of the embeddings.

Reducing the number of dimension is called manifold learning, and many different techniques exist, some of them linear, such as Principal Component Analysis (PCA), Independent Component Analysis (ICA), Linear Discriminant Analysis (LDA), and Latent Sementic Analysis / Indexing (LSA / LSI), and some are non-linear, such as Isomap, Locally Linear Embedding (LLE), Hessian Eigenmapping, Spectral embedding, Local tangent space embedding, Multi Dimensional Scaling (MDS), and t-distributed Stochastic Neighbor Embedding (t-SNE).

To display the word embedding, let us use t-SNE, a great technique adapted to high dimensional data to reveal local structures and clusters...