Text mining framework and methods

There are many different methods to use in text mining. The goal here is to provide a basic framework to apply to such an endeavor. This framework is not all-inclusive of the possible methods but will cover those that are probably the most important for the vast majority of projects that you will work on. Additionally, I will discuss the modeling methods in as succinct and clear a manner as possible because they can get quite a bit complicated. Gathering and compiling the text data is a topic that could take up several chapters. Therefore, let's begin with the assumption that the data is available from Twitter, a customer call center, scraped off the web, or whatever and is contained in some sort of text file or files.

The first task is to put the text files in one structured file referred to as a Corpus. The number of documents could be just one, dozens, hundreds, or even thousands. R can handle a number of raw text files including RSS feeds, pdf files,...