Extracting embeddings with ALBERT

With Hugging Face transformers, we can use the ALBERT model just like how we used BERT. Let's explore this with a small example. Suppose we need to get the contextual word embedding of every word in the sentence Paris is a beautiful city. Let's see how to do that with ALBERT.

Import the necessary modules:

from transformers import AlbertTokenizer, AlbertModel

Download and load the pre-trained ALBERT model and tokenizer. In this tutorial, we'll use the ALBERT-base model:

model = AlbertModel.from_pretrained('albert-base-v2')
tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')

Now, feed the sentence to the tokenizer and get the preprocessed input:

sentence = "Paris is a beautiful city" 
inputs = tokenizer(sentence, return_tensors="pt")

Let's print the inputs:

print(inputs)

The preceding code prints the following. As we can observe, our inputs consist of input_ids, token_type_ids (segment...