First example – a keyword extraction algorithm
In this section, you are going to use a phaser to implement a keyword extraction algorithm. The main purpose of these kinds of algorithms is to extract the words from a text document or a collection of documents that define the document of the document inside the collection better. These terms can be used to summarize the documents, clustering them or to improve the information search process.
The most basic algorithm to extract the keywords of the documents in a collection (but it's still commonly used nowadays) is based on the TF-IDF measure where:
- TF (short for term frequency) is the number of times that a word appears in a document.
- DF (short for document frequency) is the number of documents that contains a word. The IDF (short for inverse document frequency) measures the information that word provides to distinguish a document from others. If a word is very common, it's IDF will be low, but if the word appears in only a...
