Historical Context and Evolution of Language Models (LMs)

There are several misconceptions surrounding LMs, notably the belief that they were invented by OpenAI. However, the idea of LMs is not just a few years old, it actually spans several decades. As illustrated in figure 1.2, the concept behind some LMs is quite intuitive: given an input sequence, the task of the model is to predict the next token:

Figure 1.2: LMs and prediction of next token given the previous words (context)

To truly appreciate the sophistication of modern LMs, it's essential to explore the historical evolution and the diverse range of disciplines from which they draw inspiration, all the way up to the recent transformative developments we are currently witnessing.

Early Developments

The origins of LMs can be traced back several decades, originating in the foundational work on statistical models for natural language processing. Early LMs primarily utilized basic statistical methods, such as n-gram models...