Statistical NLU

Hand-crafted grammars are time consuming to develop and prone to errors. Considerable linguistic and engineering expertise is required to develop a grammar with good coverage and optimized performance. Moreover, the rules of a hand-crafted grammar cannot easily cope with the irregular input that is characteristic of spontaneous spoken language. For example, given the recognized words I would like a um flight from Paris to New York on Monday no Tuesday afternoon, our grammar would fail since um and no are not specified in the rules.

A statistical grammar is an alternative to a hand-crafted grammar. Statistical grammars are learned from data and involve collecting and annotating large amounts of relevant language data. Statistical grammars can cope with irregular input as they do not have to match the input exactly but rather assign probabilities indicating the extent to which a structure or a semantic interpretation matches the input. There are different types of statistical...