Deployment modes
There are different deployment modes available in Spark. These deployment modes define how Spark applications are launched, executed, and managed in diverse computing infrastructures. Based on these different deployment modes, it gets decided where the Spark driver, executor, and cluster manager will run.
The different deployment modes that are available in Spark are as follows:
- Local: In local mode, the Spark driver and executor run on a single JVM and the cluster manager runs on the same host as the driver and executor.
- Standalone: In standalone mode, the driver can run on any node of the cluster and the executor will launch its own independent JVM. The cluster manager can remain on any of the hosts in the cluster.
- YARN (client): In this mode, the Spark driver runs on the client and YARN’s resource manager allocates containers for executors on NodeManagers.
- YARN (cluster): In this mode, the Spark driver runs with the YARN application...
