Working with textual data

In the following example, we will consider the problem of separating text messages sent between cell phone users. Some of these messages are spam advertisements, and the objective is to separate these from normal communications (Almeida, Tiago A., José María G. Hidalgo, and Akebo Yamakami. Contributions to the study of SMS spam filtering: new collection and results. Proceedings of the 11th ACM symposium on Document engineering. ACM, 2011). By looking for patterns of words that are typically found in spam advertisements, we could potentially derive a smart filter that would automatically remove these messages from a user's inbox. However, while in previous chapters we were concerned with fitting a predictive model for this kind of problem, here we will be shifting focus to cleaning up the data, removing noise, and extracting features. Once these tasks are done, either simple or lower-dimensional features can be input into many of the algorithms we have already studied...