Fair Scheduler configuration

Getting ready

To go through the recipe in this section, we need Hadoop Cluster setup and running. By default, Apache Hadoop 1.x distribution uses FIFO scheduler and Hadoop 2.x uses Capacity Scheduler. In a cluster with multiple jobs, it is not good to use FIFO scheduler, as it will starve the jobs for resources and only the very first job in the queue is executed; all other jobs have to wait.

To address the preceding issue, there are two commonly used Schedulers: Fair Scheduler, and Capacity Scheduler, to allocate the cluster resources in a fair manner. In this recipe, we will see how to configure Fair Scheduler. Simply put, Fair Scheduler shares resources fairly among running jobs based on queues and weights assigned.

How to do it...

1. Connect to the master1.cyrus.com master node in the cluster and switch as user hadoop.

2. Edit the yarn-site.xml as follows:

   <property>
       <name>yarn.resourcemanager.scheduler.class</name>
       <value>org.apache...