In this chapter, we discussed the life cycle of the Spark application, Spark components, and their roles in job execution. We first created a sample application in Python and used spark-submit to run our application. We discussed how the driver process constructs the DAG and its stages, and how tasks are scheduled on a cluster. In addition, we also discussed different execution modes offered by Spark in detail. Finally, we discussed possible ways to monitor your application.
In the next chapter, we shall discuss Spark SQL in detail.
