In-context learning

In-context learning is a technique where the model generates responses based on a few examples provided in the input prompt. This method leverages the model’s pre-trained knowledge and the specific context or examples included in the prompt to perform tasks without the need for parameter updates or retraining. The general approach, detailed in Language Models are Few-Shot Learners by Brown et al. (2020), describes how the extensive pre-training of these models enables them to perform tasks and generate responses based on a limited set of examples paired with instructions embedded within prompts. Unlike traditional methods that require fine-tuning for each specific task, in-context learning allows the model to adapt and respond based on the additional context provided at inference.

Central to in-context learning is the concept of few-shot prompting, which is critical for enabling models to adapt to and perform tasks without additional training data, relying...