Opportunistic containers in Hadoop 3.x
Containers are allocated to nodes by the scheduler only when there is sufficient unallocated resources at a node. YARN guarantees that once the application master dispatches a container to a node, the execution will immediately start. The execution of a container will only be completed if there is no violation of fairness or capacity, which means until some other containers ask for preemption of resources from the node, the container is guaranteed to run to completion.
The current container execution design allows an efficient task execution but it has two primary limitations, which are as follows:
- Heartbeat delay: The Node Manager at regular intervals sends heartbeats to its resource manager and the heartbeat request also contains the resource metrics of a Node Manager. If any container running on a Node Manager finishes its execution then the information is sent as part of request in the next heartbeat, which means that the Resource Manager knows that...
