Search icon CANCEL
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Conferences
Free Learning
Arrow right icon
Arrow up icon
GO TO TOP
Programming MapReduce with Scalding

You're reading from   Programming MapReduce with Scalding A practical guide to designing, testing, and implementing complex MapReduce applications in Scala

Arrow left icon
Product type Paperback
Published in Jun 2014
Publisher
ISBN-13 9781783287017
Length 148 pages
Edition 1st Edition
Languages
Tools
Arrow right icon
Author (1):
Arrow left icon
Antonios Chalkiopoulos Antonios Chalkiopoulos
Author Profile Icon Antonios Chalkiopoulos
Antonios Chalkiopoulos
Arrow right icon
View More author details
Toc

Table of Contents (11) Chapters Close

Preface 1. Introduction to MapReduce 2. Get Ready for Scalding FREE CHAPTER 3. Scalding by Example 4. Intermediate Examples 5. Scalding Design Patterns 6. Testing and TDD 7. Running Scalding in Production 8. Using External Data Stores 9. Matrix Calculations and Machine Learning Index

Coordinating job execution


A lot of times, we need to coordinate the execution of Scalding jobs and even mix them with other applications. For example, let us assume that we have implemented two Scalding jobs and one Scala application, as shown:

class JobA (args: Args) extends Job (args) { /*pipeline*/ }
class JobB (args: Args) extends Job (args) { /*pipeline*/ }
object ScalaApp extends App { /*Application logic*/ }

To ensure that we execute the preceding tasks in a predefined order, we can implement a runner class. A runner class implemented in Scala should extend App in order to work in the imperative programming style. This means that commands are executed synchronously and sequentially.

We can use this to our advantage and coordinate the execution of MapReduce tasks, other applications, and even external system commands, such as shown in the following example code:

object ExampleRunner extends App {

  val runnerArgs = Args(args)
  val configuration = new org.apache.hadoop.conf.Configuration...
lock icon The rest of the chapter is locked
Register for a free Packt account to unlock a world of extra content!
A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.
Unlock this book and the full library FREE for 7 days
Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of
Renews at $19.99/month. Cancel anytime