Understanding where YARN fits into Hadoop
If we refer to Hadoop 1.x in the first figure of this chapter, then it is clear that the responsibilities of the JobTracker mainly included the following:
Managing the computational resources in terms of map and reduce slots
Scheduling submitted jobs
Monitoring the executions of the TaskTrackers
Restarting failed tasks
Performing a speculative execution of tasks
Calculating the Job Counters
Clearly, the JobTracker alone does a lot of tasks together and is overloaded with lots of work.
This overloading of the JobTracker led to the redesign of the JobTracker, and YARN tried to reduce the responsibilities of the JobTracker in the following ways:
Cluster resource management and Scheduling responsibilities were moved to the global Resource Manager (RM)
The application life cycle management, that is, job execution and monitoring was moved into a per-application ApplicationMaster (AM)
The Global Resource Manager is seen in the following image:
If you look at the preceding...
