First example - a keyword extraction algorithm
In this section, you are going to use a phaser to implement a keyword extraction algorithm. The main purpose of these kinds of algorithms is to extract the words from a text document or a collection of documents, which define the document or the document inside the collection, better. These terms can be used to summarize the documents, cluster them, or to improve the information search process.
The most basic algorithm to extract the keywords of the documents in a collection (but it's still commonly used nowadays) is based on the TF-IDF measure where:
- Term Frequency (TF) is the number of times that a d appears in a document.
- Document Frequency (DF) is the number of documents that contain a word. The Inverse Document Frequency (IDF) measures the information that word provides to distinguish a document from others. If a word is very common, its IDF will be low, but if the word appears in only a few documents, its IDF will be high.
The TF-IDF of the...
