Uncovering transformer improvements using only the decoder
Recall that the decoder block of the transformer focuses on an autoregressive structure. For the decoder-only transformer line of models, the task of predicting tokens autoregressively remains the same. With the removal of the encoder, the architecture has to adapt its input to accept more than one sentence, similar to what BERT does. Starting, ending, and separator tokens are used to encode input data sequentially. Masking is still performed to prevent the model from depending on the current token to predict future tokens from the input data during predictions, which is similar to the original transformer along with positional embeddings.
Diving into the GPT model family
All these architectural concepts were introduced by the GPT model in 2018, which is short for generative pre-training. As the name suggests, GPT also adopts unsupervised pre-training as the initial stage and subsequently moves into the supervised fine...
