Text-specific feature construction
Until this point, we have been working with categorical and numerical data. While our categorical data has come in the form of a string, the text has been part of a single category. We will now dive deeper into longer—form text data. This form of text data is much more complex than single—category text, because we now have a series of categories, or tokens.
Before we get any further into working with text data, let's make sure we have a good understanding of what we mean when we refer to text data. Consider a service like Yelp, where users write up reviews of restaurants and businesses to share their thoughts on their experience. These reviews, all written in text format, contain a wealth of information that would be useful for machine learning purposes, for example, in predicting the best restaurant to visit.
In general, a large part of how we communicate in today's world is through written text, whether in messaging services, social media, or email. As...
