Spark application in distributed-mode

A Spark application is made of up of Driver and Executor(s) processes. Each application in Spark contains one driver process and one or more executor processes. The driver is the central coordinator of the application that drives the application. Spark Driver communicates and divides work among one or more executors. In distributed mode, Spark driver and each executor runs in separate JVM.

Logical Representation of a Spark Application in Distributed Mode

Driver program

SparkContext is initialized in the Driver JVM. Spark driver can be considered as the master of Spark applications. The following are the responsibilities of Spark Driver program:

• It creates the physical plan of execution...