TDD for Scalding developers

This section describes an approach to deal with the development process with a testing context. We will work through code examples in Scalding, but the majority of the concepts are also valid in a broader context, in other Scala-based MapReduce frameworks such as Scoobi and Scrunch in particular. We will first describe the testing strategy, and then implement a framework to support this strategy.

Implementing the TDD methodology

The rest of the chapter presents a step-by-step implementation of the TDD methodology. We will use the example presented in the previous chapter, which requires the following actions:

• Read a logfile and count the number of visits per user per day, where the logfiles contain user URL requests

• Enrich user information with the e-mail and address retrieved from an additional input file containing user info, which is indexed via the user ID

Once the exploration of the data and algorithm design is complete, the test-driven methodology requires the...