The architecture of Spark
In this section, we will discuss the architecture of Spark and its various components in detail. We will also briefly talk about the various extensions/libraries of Spark, which are developed over the core Spark framework.
Spark is a general-purpose computing engine that initially focused to provide solutions to the iterative and interactive computations and workloads. For example, machine learning algorithms, which reuse intermediate or working datasets across multiple parallel operations.
The real challenge with iterative computations is the dependency of the intermediate data/steps on the overall job. This intermediate data needs to be cached in the memory itself for faster computations because flushing and reading from a disk is an overhead, which, in turn, makes the overall process unacceptably slow.
The creators of Apache Spark not only provided scalability, fault tolerance, performance, and distributed data processing, but also provided in-memory processing...
