Introducing LLMs
In this section, we’ll take a more systematic approach and dive deeper into transformer-based architectures. As we mentioned in the introduction, the transformer block has changed remarkedly little since its introduction in 2017. Instead, the main advances have come in terms of larger models and larger training sets. For example, the original GPT model (GPT-1) has 117M parameters, while GPT-3 (Language Models are Few-Shot Learners, https://arxiv.org/abs/2005.14165) has 175B, a thousandfold increase. We can distinguish two informal transformer model categories based on size:
- Pre-trained language models (PLMs): Transformers with fewer parameters, such as Bidirectional Encoder Representations from Transformers (BERT) and generative pre-trained transformers (GPT), fall into this category. Starting with BERT, these transformers introduced the two-step pre-training/FT paradigm. The combination of the attention mechanism and unsupervised pre-training (masked...
