Calculating the frequency distributions of words

A frequency distribution counts the number of occurrences of distinct data values. These are of value as we can use them to determine which words or phrases within a document are most common, and from that infer those that have greater or lesser value.

Frequency distributions can be calculated using several different techniques. We will examine them using the facilities built into NLTK.

How to do it

NLTK provides a class, ntlk.probabilities.FreqDist, that allow us to very easily calculate the frequency distribution of values in a list. Let's examine using this class (code is in 07/freq_dist.py):

1. To create a frequency distribution using NLTK, start by importing the...