Probabilistic framework

When building AI-intensive applications that interact with LLMs, you will likely come across API parameters relating to probabilities of tokens. To understand how LLMs relate to the concept of probabilities, this section introduces the probabilistic framework underpinning language models.

Language modeling is typically done with a probabilistic view in mind, rather than in absolute and deterministic terms. This allows the algorithms to deal with the uncertainty and ambiguity often found in natural language.

To build an intuitive understanding of probabilistic language modeling, consider the following start of a sentence, for which you want to predict the next word:

The

This is obviously an ambiguous task with many possible answers. The article the is a very common and generic word in the English language, and the possibilities are endless. Any noun, such as house, dog, spoon, etc. could be a valid possible continuation of the sentence. Even adjectives...