An introduction to schedulers

The scheduler is responsible for providing resources to different tasks of running applications. It is only responsible for scheduling of tasks and is not concerned with status tracking and monitoring of tasks. The scheduler ensures meeting resource requirements in terms of memory, cores, disk, and network for the application. At granular level, it meets the resource requirement of containers running for the particular application. The default scheduler of Hadoop uses a single queue (root queue) to accept and schedule applications. It means that all the applications are submitted to the root queue.

You can view the details of the configured scheduler through ResourceManager web UI at http://<ResourceManager IP>:8088/cluster/scheduler. This is shown in the following screenshot:

YARN provides interfaces for implementation of pluggable scheduler. The two popular schedulers available with Hadoop are:

• Fair scheduler

• Capacity scheduler

Fair scheduler

Fair scheduler...