High-level changes from MRv1 to MRv2

With the introduction of YARN, the architecture for Hadoop job execution and management framework changed. In this section, we'll discuss the list of high-level changes observed in MRv2 framework.

The evolution of the MRApplicationMaster service

In YARN, the responsibility of JobTracker is divided across the ResourceManager service and application-specific ApplicationMaster service. For management of MapReduce jobs, MRApplicationMaster service is defined in the Hadoop framework. For each MapReduce job submitted to ResourceManager, an instance MRApplicationMaster service is launched. After successful execution of the job, the MRApplicationMaster service is terminated.

The MRApplicationMaster service is responsible for:

• Registering the job with the ResourceManager

• Negotiating YARN containers for execution of map reduce tasks

• Interacting with NodeManager to manage execution of allocated containers

• Handling task failure and reinitiate failed tasks

• Handling client...