Using slim JAR files

The majority of articles and tutorials recommend developers to package all the dependencies and the application code into a single JAR file. This is known as the fat jar approach and can be achieved using maven or sbt.

A build tool can generate the JAR file with a single command, such as mvn package, once we have all the appropriate plugins, such as maven-assembly-plugin, in place.

This process is awesome, until we have to deal with the compiler or the deployment process more than once a day. Assembling a single distributable archive takes time. The plugin needs to iterate through all the project dependencies, uncompressing every single dependency and aggregating the project output along with its dependencies, modules, and other files.

Scalding applications depend on Hadoop libraries, the Scala library, Cascading libraries, and other utility libraries. The dependency hierarchy means that the resulting JAR files occupy between 60 MB and 100 MB, depending on the amount of...