Search icon CANCEL
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Conferences
Free Learning
Arrow right icon
Arrow up icon
GO TO TOP
Learning Hadoop 2

You're reading from   Learning Hadoop 2 Design and implement data processing, lifecycle management, and analytic workflows with the cutting-edge toolbox of Hadoop 2

Arrow left icon
Product type Paperback
Published in Feb 2015
Publisher Packt
ISBN-13 9781783285518
Length 382 pages
Edition 1st Edition
Tools
Arrow right icon
Toc

Table of Contents (13) Chapters Close

Preface 1. Introduction 2. Storage FREE CHAPTER 3. Processing – MapReduce and Beyond 4. Real-time Computation with Samza 5. Iterative Computation with Spark 6. Data Analysis with Apache Pig 7. Hadoop and SQL 8. Data Lifecycle Management 9. Making Development Easier 10. Running a Hadoop Cluster 11. Where to Go Next Index

Running the examples

The source code of all examples is available at https://github.com/learninghadoop2/book-examples.

Gradle (http://www.gradle.org/) scripts and configurations are provided to compile most of the Java code. The gradlew script included with the example will bootstrap Gradle and use it to fetch dependencies and compile code.

JAR files can be created by invoking the jar task via a gradlew script, as follows:

./gradlew jar

Jobs are usually executed by submitting a JAR file using the hadoop jar command, as follows:

$ hadoop jar example.jar <MainClass> [-libjars $LIBJARS] arg1 arg2 … argN

The optional -libjars parameter specifies runtime third-party dependencies to ship to remote nodes.

Note

Some of the frameworks we will work with, such as Apache Spark, come with their own build and package management tools. Additional information and resources will be provided for these particular cases.

The copyJar Gradle task can be used to download third-party dependencies into build/libjars/<example>/lib, as follows:

./gradlew copyJar

For convenience, we provide a fatJar Gradle task that bundles the example classes and their dependencies into a single JAR file. Although this approach is discouraged in favor of using –libjar, it might come in handy when dealing with dependency issues.

The following command will generate build/libs/<example>-all.jar:

$ ./gradlew fatJar
lock icon The rest of the chapter is locked
Register for a free Packt account to unlock a world of extra content!
A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.
Unlock this book and the full library FREE for 7 days
Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of
Renews at $19.99/month. Cancel anytime