Accessing text from diverse sources

Reading data from diverse sources for analysis, and exporting the results to another system for reporting purposes can be a daunting task that can sometimes take even more time than the real analysis. There are various sources from which we can gather text; some of them are HTML pages, social media, RSS feeds, JSON or XML, enterprise environments, and so on. The source has a very important role to play in the quality of textual data and the way we access the source. For instance, in the case of an enterprise environment, the common sources of text or data can be database and log files. In a web ecosystem, web pages are the source of data. When we consider web service applications, the sources can be JSON or XML over HTTP or HTTPS. We will look into various data sources and ways in which we can collect data from them.

File system

Reading from a file system is a very basic capability that any programming language should provide. We may have collections of...