Monitoring Scalding jobs

A web application that helps us visualize the operational details around all phases of our Scalding applications such as application development, debugging, performance tuning, and operator monitoring is Driven. This application is developed by Concurrent, the same company that developed and open sourced Cascading.

Driven (http://driven.cascading.io/) is a free cloud service that receives and visualizes telemetry data from running Scalding applications.

To enable this, we need to include the following plugin:

<dependency>
  <groupId>driven</groupId>
  <artifactId>driven-plugin</artifactId>
  <version>1.0-eap-59</version>
  <classifier>io</classifier>
</dependency>

Then, create a new account and receive an appropriate API key that can be defined as a system variable, as shown:

$ export DRIVEN_API_KEY=D991A15E7A174E098900CDEE4F3A3CA6
$ hadoop jar ...

Driven provides both high-level and low-level representations...