Identify features and uses for language modeling
As we saw in the NLP scenarios section, language modeling is one of the tasks that can be provided by NLP as part of the language area of AI.
Language modeling is based on the concept of prediction; the next word in a sequence of words is predicted by the model based on the “preceding words’ context.”
Language modeling requires the construction of a probabilistic model for natural language (NL). This model converts sequences of words into “probabilities,” estimating the probability that a certain word will occur based on how words precede or follow it. One of the most prevalent approaches to language modeling is through n-grams or more advanced techniques, such as recurrent neural networks (RNNs) and transformer models.
In the following subsections, you will look at two of the core capabilities of a language model: to “understand conversations” and to “answer questions...
