We have covered enough information about YARN in previous chapters. In this section, we will talk about the execution of MapReduce over YARN. The JobTracker in Hadoop version 1 has a bottleneck due to a scalability limit of 4,000 nodes. Yahoo realizes that their current requirement needs a scaling of up to 20,000 nodes. The latter was certainly not possible due to the legacy architecture of the job tracker. Yahoo then introduced YARN, which broke the function of the job tracker for efficient management. We covered the detail architecture in Chapter 3, YARN Resource Management in Hadoop.
The node manager in YARN has enough memory to launch multiple containers. The application master can request any number of containers from the resource manager, which keeps track of the available resources in the YARN cluster. The job type is not limited to MapReduce...
