Extracting Features from Text Variables

Text can be one of the variables in our datasets. For example, in insurance, information describing the circumstances of an incident can come from free text fields in a form. If a company gathers customer reviews, this information will be collected as short pieces of text provided by the users. Text data does not show the tabular pattern of the datasets that we have worked with throughout this book. Instead, information in texts can vary in length and content, as well as writing style. We can extract a lot of information from text variables to use as predictive features in machine learning models. The techniques we will cover in this chapter belong to the realm of Natural Language Processing (NLP). NLP is a subfield of linguistics and computer science. It is concerned with the interactions between computer and human language, or, in other words, how to program computers to understand human language. NLP includes a multitude of techniques to understand...