Understanding BERT

BERT (Bidirectional Encoder Representation from Transformers) is a Transformer model among a plethora of Transformer models that have come to light over the past few years.

BERT was introduced in the paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding by Delvin et al. (https://arxiv.org/pdf/1810.04805.pdf). The Transformer models are divided into two main factions:

• Encoder-based models
• Decoder-based (autoregressive) models

In other words, either the encoder or the decoder part of the Transformer provides the foundation for these models, compared to using both the encoder and the decoder. The main difference between the two is how attention is used. Encoder-based models use bidirectional attention, whereas decoder-based models use autoregressive (that is, left to right) attention.

BERT is an encoder-based Transformer model. It takes an input sequence (a collection of tokens) and produces an encoded...