We will browse through the links provided in the following sections to build up a list of sources to harness interesting data in usable formats. However, this list is not at all exhaustive.

Some of these sources have an Application Programming Interface (API) that allows more sophisticated access to interesting data. An API specifies the interactions and defines how data is communicated.

We will first create an input.txt text file with a couple of lines of text to be read by the program. We keep this file in an easy-to-access directory because it will be referenced later. For example, the text file we're dealing with contains a seven-line quote by Plato. Here's what our terminal prints when we issue the following command:

$ cat input.txt

And how will you inquire, Socrates,
into that which you know not? 
What will you put forth as the subject of inquiry? 
And if you find what you want, 
how will you ever know that 
this is what you did not know?

Create a new file to start coding. We call our file Main.hs.

Haskell provides useful input and output (I/O) capabilities for reading input and writing output in different ways. In our case, we use readFile to specify a path of a file to be read. Using the do keyword in main suggests that we are joining several IO actions together. The output of readFile is an I/O string, which means it is an I/O action that returns a String type.

Now we're about to get a bit technical. Pay close attention. Alternatively, smile and nod. In Haskell, the I/O data type is an instance of something called a Monad. This allows us to use the <- notation to draw the string out of this I/O action. We then make use of the string by feeding it into our countWords function that counts the number of words in each line. Notice how we separated the countWords function apart from the impure main function.

Finally, we print the output of countWords. The $ notation means we are using a function application to avoid excessive parenthesis in our code. Without it, the last line of main would look like print (countWords input).

For simplicity's sake, this code is easy to read but very fragile. If an input.txt file does not exist, then running the code will immediately crash the program. For example, the following command will generate the error message:

$ runhaskell Main.hs

Main.hs: input.txt: openFile: does not exist…

To make this code fault tolerant, refer to the Catching I/O code faults recipe.

Create a new file, name it Main.hs, and perform the following steps:

This recipe demonstrates two ways to catch errors, listed as follows:

Conveniently, Haskell also comes with a doesFileExist :: FilePath -> IO Bool function from the System.Directory module. We can simplify the preceding code by modifying the input <- … line. It can be replaced with the following snippet of code:

exists <- doesFileExist filename
input <- if exists then readFile filename else return ""

In this case, the code reads the file as an input only if it exists. Do not forget to add the following import line at the top of the source code:

import System.Directory (doesFileExist)

Prepare a simple CSV file with a list of names and their corresponding ages. This can be done using a text editor or by exporting from a spreadsheet, as shown in the following figure:

The raw input.csv file contains the following text:

$ cat input.csv 

name,age
Alex,22
Anish,22
Becca,23
Jasdev,22
John,21
Jonathon,21
Kelvin,22
Marisa,19
Shiv,22
Vinay,22

The code also depends on the csv library. We may install the library through Cabal using the following command:

$ cabal install csv

The CSV data structure in Haskell is represented as a list of records. Record is merely a list of Fields, and Field is a type synonym for String. In other words, it is a collection of rows representing a table, as shown in the following figure:

The parseCSV library function returns an Either type, with the Left side being a ParseError and the Right side being the list of lists. The Either l r data type is very similar to the Maybe a type which has the Just a or Nothing constructor.

We use the either function to handle the Left and Right cases. The Left case handles the error, and the Right case handles the actual work to be done on the data. In this recipe, the Right side is a Record. The fields in Record are accessible through any list operations such as head, last, !!, and so on.

Install the aeson library from hackage using Cabal.

Prepare an input.json file representing data about a mathematician, such as the one in the following code snippet:

$ cat input.json

{"name":"Gauss", "nationality":"German", "born":1777, "died":1855}

We will be parsing this JSON and representing it as a usable data type in Haskell.

Aeson takes care of the complications in representing JSON. It creates native usable data out of a structured text. In this recipe, we use the .: and .:? functions provided by the Data.Aeson module.

As the Aeson package uses ByteStrings instead of Strings, it is very helpful to tell the compiler that characters between quotation marks should be treated as the proper data type. This is done in the first line of the code which invokes the OverloadedStrings language extension.

We use the decode function provided by Aeson to transform a string into a data type. It has the type FromJSON a => B.ByteString -> Maybe a. Our Mathematician data type must implement an instance of the FromJSON typeclass to properly use this function. Fortunately, the only required function for implementing FromJSON is parseJSON. The syntax used in this recipe for implementing parseJSON is a little strange, but this is because we're leveraging applicative functions and lenses, which are more advanced Haskell topics.

The .: function has two arguments, Object and Text, and returns a Parser a data type. As per the documentation, it retrieves the value associated with the given key of an object. This function is used if the key and the value exist in the JSON document. The :? function also retrieves the associated value from the given key of an object, but the existence of the key and value are not mandatory. So, we use .:? for optional key value pairs in a JSON document.

If the implementation of the FromJSON typeclass is too involved, we can easily let GHC automatically fill it out using the DeriveGeneric language extension. The following is a simpler rewrite of the code:

{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE DeriveGeneric #-}
import Data.Aeson
import qualified Data.ByteString.Lazy as B
import GHC.Generics

data Mathematician = Mathematician { name :: String
                                   , nationality :: String
                                   , born :: Int
                                   , died :: Maybe Int
                                   } deriving Generic

instance FromJSON Mathematician

main = do
  input <- B.readFile "input.json"
  let mm = decode input :: Maybe Mathematician
  case mm of
    Nothing -> print "error parsing JSON"
    Just m -> (putStrLn.greet) m
    
greet m = (show.name) m ++" was born in the year "++ (show.born) m

Although Aeson is powerful and generalizable, it may be an overkill for some simple JSON interactions. Alternatively, if we wish to use a very minimal JSON parser and printer, we can use Yocto, which can be downloaded from http://hackage.haskell.org/package/yocto.

We will first set up an XML file called input.xml with the following values, representing an e-mail thread between Databender and Princess on December 18, 2014 as follows:

$ cat input.xml

<thread>
    <email>
        <to>Databender</to>
        <from>Princess</from>
        <date>Thu Dec 18 15:03:23 EST 2014</date>
        <subject>Joke</subject>
        <body>Why did you divide sin by tan?</body>
    </email>
    <email>
        <to>Princess</to>
        <from>Databender</from>
        <date>Fri Dec 19 3:12:00 EST 2014</date>
        <subject>RE: Joke</subject>
        <body>Just cos.</body>
    </email>
</thread>

Using Cabal, install the HXT library which we use for manipulating XML documents:

$ cabal install hxt

The library function, runX, takes in an Arrow. Think of an Arrow as a more powerful version of a Monad. Arrows allow for stateful global XML processing. Specifically, the runX function in this recipe takes in IOSArrow XmlTree String and returns an IO action of the String type. We generate this IOSArrow object using the readString function, which performs a series of operations to the XML data.

For a deep insight into the XML document, //> should be used whereas /> only looks at the current level. We use the //> function to look up the date attributes and display all the associated text.

As defined in the documentation, the hasName function tests whether a node has a specific name, and the getText function selects the text of a text node. Some other functions include the following:

All the Arrow functions can be found on the Text.XML.HXT.Arrow.XmlArrow documentation at http://hackage.haskell.org/package/hxt/docs/Text-XML-HXT-Arrow-XmlArrow.html.

We will be extracting the values from an HTML table, so start by creating an input.html file containing a table as shown in the following figure:

The HTML behind this table is as follows:

$ cat input.html

<!DOCTYPE html>
<html>
    <body>
        <h1>Course Listing</h1>
        <table>
            <tr>
                <th>Course</th>
                <th>Time</th>
                <th>Capacity</th>
            </tr>
            <tr>
                <td>CS 1501</td>
                <td>17:00</td>
                <td>60</td>
            </tr>
            <tr>
                <td>MATH 7600</td>
                <td>14:00</td>
                <td>25</td>
            </tr>
            <tr>
                <td>PHIL 1000</td>
                <td>9:30</td>
                <td>120</td>
            </tr>
        </table>
    </body>
</html>

If not already installed, use Cabal to set up the HXT library and the split library, as shown in the following command lines:

$ cabal install hxt
$ cabal install split

This is very similar to XML parsing, except we adjust the options of readString to [withParseHTML yes, withWarnings no].

We will be collecting all links from a Wikipedia web page. Make sure to have an Internet connection before running this recipe.

Install the HandsomeSoup CSS selector package, and also install the HXT library if it is not already installed. To do this, use the following commands:

$ cabal install HandsomeSoup
$ cabal install hxt

The HandsomeSoup package allows easy CSS selectors. In this recipe, we run the #bodyContent a selector on a Wikipedia article web page. This finds all link tags that are descendants of an element with the bodyContent ID.

Another common way to obtain data online is through POST requests. To find out more, refer to the Learning how to perform HTTP POST requests recipe.

For this recipe, access to the Internet is necessary.

Install the HandsomeSoup CSS selector package, and also install the HXT library if it is not already installed:

$ cabal install HandsomeSoup
$ cabal install hxt

Running the code will display all search results relating to "poon", such as "Poonam" or "Witherspoon".

A POST request needs the specified URI, headers, and body. By filling out a Request data type, it can be used to establish a server request.

Refer to the Understanding how to perform HTTP GET requests recipe for details on how to perform a GET request instead.

Make sure to have a strong Internet connection.

Install the hxt and HandsomeSoup packages using Cabal:

$ cabal install hxt
$ cabal install HandsomeSoup

The code starts by searching for "a" in the directory lookup. This will most likely fault in an error as there are too many results. So, in the next iteration, the code will refine its search by querying for "aa", then "aaa", until there is no longer TooManyResultsErr :: SearchResultErr.

Then, it will enumerate to the next logical search query "aab", and if that produces no result, it will search for "aac", and so on. This brute force prefix search will obtain all items in the database. We can gather the mass of data, such as names and department types, to perform interesting clustering or analysis later on. The following figure shows how the program starts:

We need to install MongoDB on our local machine and have a database instance running in the background while we run the code in this recipe.

MongoDB installation instructions are located at http://www.mongodb.org. On Debian-based operating systems, we can use apt-get to install MongoDB, using the following command line:

$ sudo apt-get install mongodb

Run the database daemon by specifying the database file path as follows:

$ mkdir ~/db
$ mongod --dbpath ~/db

Fill up a "people" collection with dummy data as follows:

$ mongo
> db.people.insert( {first: "Joe", last: "Shmoe"} )

Install the MongoDB package from Cabal using the following command:

$ cabal install mongoDB

A pipe is established by the driver between the running program and the database. This allows running MongoDB operations to bridge the program with the database. The find function takes a query, which we construct by evoking the select :: Selector -> Collection -> aQueryOrSelection function.

Other functions can be found in the documentation at http://hackage.haskell.org/package/mongoDB/docs/Database-MongoDB-Query.html.

If the MongoDB database is on a remote server, refer to the Reading from a remote MongoDB server recipe to set up a connection with remote databases.

We should create a remote database. MongoLab (https://mongolab.com) and MongoHQ (http://www.mongohq.com) offer MongoDB as a service and have free options to set up a small development database.

Install the MongoDB package from Cabal as follows:

$ cabal install mongoDB

Also, install the helper following helper libraries as follows:

$ cabal install split
$ cabal install uri

If the database is on a local machine, refer to the Using MongoDB queries in Haskell recipe.

We need to install the SQLite database if it isn't already set up. It can be obtained from http://www.sqlite.org. On Debian systems, we can get it from apt-get using the following command:

$ sudo apt-get install sqlite3

Now create a simple database to test our code, using the following commands:

$ sqlite3 test.db "CREATE TABLE test \
(id INTEGER PRIMARY KEY, str text); \
INSERT INTO test (str) VALUES ('test string');"

We must also install the SQLite Haskell package from Cabal as follows:

$ cabal install sqlite-simple

This recipe will dissect the example code presented on the library's documentation page available at http://hackage.haskell.org/package/sqlite-simple/docs/Database-SQLite-Simple.html.