RNN and LSTM networks are widely used in sequential tasks such as next word prediction, machine translation, text generation, and more. However one of the major challenges with the recurrent model is capturing the long-term dependency.

To overcome this limitation of RNNs, a new architecture called Transformer was introduced in the paper Attention Is All You Need. The transformer is currently the state-of-the-art model for several NLP tasks. The advent of the transformer created a major breakthrough in the field of NLP and also paved the way for new revolutionary architectures such as BERT, GPT-3, T5, and more.

The transformer model is based entirely on the attention mechanism and completely gets rid of recurrence. The transformer uses a special type of attention mechanism called self-attention. We will learn about this in detail in the upcoming sections.

Let's understand how the transformer works with a language translation task. The transformer...

The transformer consists of a stack of number of encoders. The output of one encoder is sent as input to the encoder above it. As shown in the following figure, we have a stack of number of encoders. Each encoder sends its output to the encoder above it. The final encoder returns the representation of the given source sentence as output. We feed the source sentence as input to the encoder and get the representation of the source sentence as output:

Note that in the transformer paper Attention Is All You Need, the authors have used , meaning that they stacked up six encoders one above the another. However, we can try out different values of . For simplicity and better understanding, let's keep :

Okay, the question is how exactly does the encoder work? How is it generating the representation for the given source sentence (input sentence)...

Suppose we want to translate the English sentence (source sentence) I am good to the French sentence (target sentence) Je vais bien. To perform this translation, we feed the source sentence I am good to the encoder. The encoder learns the representation of the source sentence. In the previous section, we learned how exactly the encoder learns the representation of the source sentence. Now, we take this encoder's representation and feed it to the decoder. The decoder takes the encoder representation as input and generates the target sentence Je vais bien, as shown in the following figure:

In the encoder section, we learned that, instead of having one encoder, we can have a stack of encoders. Similar to the encoder, we can also have a stack of decoders. For simplicity, let's set . As shown in the following figure, the output of one decoder is sent as the input to the decoder...

To give more clarity, the complete transformer architecture with the encoder and decoder is shown in the following figure:

In the preceding figure, Nx denotes that we can stack number of encoders and decoders. As we can observe, once we feed the input sentence (source sentence), the encoder learns the representation and sends the representation to the decoder, which in turn generates the output sentence (target sentence).

We can train the transformer network by minimizing the loss function. Okay, but what loss function should we use? We learned that the decoder predicts the probability distribution over the vocabulary and we select the word that has the highest probability as output. So, we have to minimize the difference between the predicted probability distribution and the actual probability distribution. First, how can we find the difference between the two distributions? We can use cross-entropy for that. Thus, we can define our loss function as a cross-entropy loss and try to minimize the difference between the predicted and actual probability distribution. We train the network by minimizing the loss function and we use Adam as an optimizer.

One additional point we need to note down is that to prevent overfitting, we apply dropout to the output of each sublayer and we also apply dropout to the sum of the embeddings and the positional encoding.

Thus, in this chapter, we learned...

We started off the chapter by understanding what the transformer model is and how it uses encoder-decoder architecture. We looked into the encoder section of the transformer and learned about different sublayers used in encoders, such as multi-head attention and feedforward networks.

We learned that the self-attention mechanism relates a word to all the words in the sentence to better understand the word. To compute self-attention, we used three different matrices, called the query, key, and value matrices. Following this, we learned how to compute positional encoding and how it is used to capture the word order in a sentence. Next, we learned how the feedforward network works in the encoder and then we explored the add and norm component.

After understanding the encoder, we understood how the decoder works. We explored three sublayers used in the decoder in detail, which are the masked multi-head attention, encoder-decoder attention, and feedforward network. Following this...

Let's put our newly acquired knowledge to the test. Try answering the following questions:

1. What are the steps involved in the self-attention mechanism?
2. What is scaled dot product attention?
3. How do we create the query, key, and value matrices?
4. Why do we need positional encoding?
5. What are the sublayers of the decoder?
6. What are the inputs to the encoder-decoder attention layer of the decoder?

To learn more, check out the following resources:

• Attention Is All You Need by Ashish Vaswani, Noam Shazeer, and Niki Parmar, available at https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf
• The Illustrated Transformer blog by Jay Alammar, at http://jalammar.github.io/illustrated-transformer