Creating a pipeline to search text

Searching is a rich and complex topic. There are many different types of searches and approaches to perform a search. The intent here is to demonstrate how various NLP techniques can be applied to support this effort. A single text document can be processed at one time in a reasonable time period on most machines. However, when multiple large documents need to be searched, then creating an index is a common approach to support searches. This results in a search process that completes in a reasonable period of time. We will demonstrate one approach to create an index and then search using the index. Although the text we will use is not that large, it is sufficient to demonstrate the process. We need to do the following:

• Read the text from the file
• Tokenize and find sentence boundaries
• Remove stop words
• Accumulate the index statistics
• Write out the index file

There are several factors that influence the contents of an index file, including:

• Removal of stop words...