Before getting started with the examples, we will set the system up with NLTK and other dependent Python libraries. The pip installer can be used to install NLTK, with an optional installation of numpy, as follows:

sudo pip install -U nltk
sudo pip install -U numpy

The NLTK corpora and various modules can be installed by using the common NLTK downloader in the Python interactive shell or a Jupyter Notebook, shown as follows:

import nltk
nltk.download()

The preceding command will open an NLTK Downloader, as follows. Select the packages or collections that are required:

As shown in the preceding screenshot, specific collections, text corpora, NLTK models, or packages, can be selected and installed. Navigate to stopwords and install it for future use. The following is a list of modules that are required for this chapter's examples:

First, we will provide a hands-on overview of NLTK by working on some basic NLP tasks, such as text preprocessing and exploratory analysis. The text preprocessing step involves tasks such as tokenization, stemming, and stop word removal. An exploratory analysis of prepared text data can be performed to understand its main characteristics, such as the main topic of the text and word frequency distributions.

Word tokens are the basic units of text involved in any NLP task. The first step, when processing text, is to split it into tokens. NLTK provides different types of tokenizers for doing this. We will look at how to tokenize Twitter comments from the Twitter samples...

Once we have the tokenized data, one of the basic analyses that is commonly performed is counting words or tokens and their distributions in the document. This will enable us to know more about the main topics in the document. Let's start by analyzing the web text data that comes with NLTK:

>>> import nltk
>>> from nltk.corpus import webtext
>>> webtext_sentences = webtext.sents('firefox.txt')
>>> webtext_words = webtext.words('firefox.txt')
>>> len(webtext_sentences)
1142
>>> len(webtext_words)
102457

Note that we have only loaded the text related to the Firefox discussion forum (firefox.txt), though the web text data has other data, as well (like advertisements and movie script text). The preceding code output gives the number of sentences and words, respectively, in the entire...

We have analyzed some of the basic NLP preprocessing tasks, such as tokenization, stemming, and stop word removal. We have also explored how to determine and visualize word distribution in a text corpus. In this section, we will get deeper into NLTK by looking at POS tagging.

POS refers to categorizing the words in a sentence into specific syntactic or grammatical functions. In English, the main parts of speech are nouns, pronouns, adjectives, verbs, adverbs, prepositions, determiners, and conjunctions. POS tagging is the task of attaching one of these categories to each of the words or tokens in a text. NLTK provides both a set of tagged text corpus and a set of POS trainers for creating custom...

We will now look at classifying sentiments in the movie reviews corpus in NLTK. The complete Jupyter Notebook for this example is available at Chapter02/01_example.ipynb, in the book's code repository.

First, we will load the movie reviews based on the sentiment categories, which are either positive or negative, using the following code:

cats = movie_reviews.categories()
reviews = []
for cat in cats:
    for fid in movie_reviews.fileids(cat):
        review = (list(movie_reviews.words(fid)),cat)
        reviews.append(review)
random.shuffle(reviews)

The categories() function returns either pos or neg, for positive and negative sentiments, respectively. There are 1,000 reviews in each of the positive and negative categories. We use the Python random.shuffle() function to convert the grouped positive and negative reviews into a random...

In the previous section, we utilized simple binary features for the words in the reviews in order to learn positive and negative sentiments. A better approach would be to use latent features, such as the frequency of the words used in the text. Compared to a binary representation of the presence or absence of words, the count of the words may better capture the characteristics of the text or document. Bag-of-words is a vector representation of text. Each of the vector dimensions captures either the frequency, presence or absence, or weighted values of words in the text. A bag-of-words representation does not capture the order of the words.

The binary feature extraction that was discussed in the previous section is, therefore, a simple bag-of-words representation of text. We will now look at an example of classifying sentiments in tweets using...

In this chapter, we covered common NLP tasks, such as preprocessing and exploratory analysis of text using the NLTK library. The unstructured characteristics of real-world data need extensive preprocessing, such as tokenization, stemming, and stop word removal, to make it suitable for ML. As you saw in the examples, NLTK provides a very extensive API for carrying out these preprocessing steps. It provides built-in packages and modules, and supports flexibility to build custom modules, such as user-defined stemmers and tokenizers.

We also discussed using NLTK for POS tagging, which is another common NLP task, used for issues such as word sense disambiguation and answering questions. Applications such as sentiment classification are widely used for their research and business value. We covered some basic examples of text classification, in the context of sentiment analysis...