Emergent abilities of LLMs

In this section, we’ll discuss the phenomenon of emergent abilities of LLMs, first summarized in https://arxiv.org/abs/2206.07682. The paper defines emergent abilities as follows:

An ability is emergent if it is not present in smaller models but is present in larger models.

These abilities represent a qualitative difference between large and small language models, which cannot be predicted by extrapolation.

We’ll start with the ability known as few-shot prompting (or in-context learning), popularized by GPT-3. Here, the initial user prompt is an instruction the LLM has to follow through its response without any additional training. The prompt itself may describe with natural text one or more training examples (hence, the term few-shot). This is the only context that the LLM can use for training before generating its response. The following diagram shows an example of a few-shot prompt:

Figure 8.15 –...