Spark

Hadoop has been used as a processing framework for large datasets for the past decade and it has brought tremendous value and cost saving to organizations. MapReduce has evolved over a time but it is not efficient for a few use cases like near real-time computation, multi-pass computation, which is iterative processing, and so on. Every time the data is processed, it has to be written into the disk and then you have to pick data from disk for further processing. Along with this, if we need to add additional use cases which require libraries such as Mahout and Apache Storm, then it has to be integrated separately in the Hadoop cluster. Spark is a distributed data processing framework that provides functional APIs for manipulating data at scale, in-memory data caching, and reusability of datasets. Spark utilizes the concept of the direct acyclic graph (DAG), which is a data lineage graph that helps in recomputing tasks in case of failure. Spark supports a number of file formats and rich...