Understanding the Transformer model

Automatic language translation is an important research topic in NLP. In recent years, sequence-to-sequence models, such as recurrent neural networks (RNNs), long short-term memory (LSTM), and gated recurrent units (GRUs) have been proven effective [2]. The sequence “I just love eating Doritos” in English can be translated instantly by a sequence-to-sequence model to “Ich liebe es einfach, Doritos zu essen” in German. RNNs, LSTM, and GRUs suffer from limitations in capturing long-range dependencies in sequential data due to their short-term memory and sequential processing nature. The need for sequential computation results in long computational time as well. The Transformer architecture addresses these weaknesses through a self-attention mechanism. The self-attention mechanism in a Transformer is like looking at each word and deciding how much attention it should give to the other words. It enables the Transformer to...