What a tokenizer is and what it does

The first step in feature engineering text data is to decide on the tokenization of the text. The tokenization of text is a process of extracting parts of words that capture the meaning of the text without too many extra details.

There are different ways to extract tokens, which we’ll explore in this chapter, but to illustrate the problem of extracting tokens, let’s look at one word that can take different forms – print. The word by itself can be a token, but it can be in different forms, such as printing, printed, printer, prints, imprinted, and many others. If we use a simple tokenizer, each of these words will be one token – which means quite a few tokens. However, all these tokens capture some sort of meaning related to printing, so maybe we do not need so many of them.

This is where tokenizers come in. Here, we can decide how to treat these different forms of the word. We could take the main part only –...