The bag-of-words strategy

In NLP, a very common pipeline can be subdivided into the following steps:

1. Collecting a document into a corpus.
2. Tokenizing, stopword (articles, prepositions and so on) removal, and stemming (reduction to radix-form).
3. Building a common vocabulary.
4. Vectorizing the documents.
5. Classifying or clustering the documents.

The pipeline is called bag-of-words and will be discussed in this chapter. A fundamental assumption is that the order of each single word in a sentence is not important. In fact, when defining a feature vector, as we're going to see, the measures taken into account are always related to frequencies and therefore they are insensitive to the local positioning of all elements. From some viewpoints, this is a limitation because in a natural language the internal order of a sentence is necessary to preserve the meaning; however, there are many...