Simplifying the Lambda Architecture using Delta Lake

A typical Lambda Architecture has three major components: a batch layer, a streaming layer, and a serving layer. In Chapter 2, Data Ingestion, you were able to view an implementation of the Lambda Architecture using Apache Spark's unified data processing framework. The Spark DataFrames API, Structured Streaming, and SQL engine help to make Lambda Architecture simpler. However, multiple data storage layers are still required to handle batch data and streaming data separately. These separate data storage layers could be easily consolidated by using the Spark SQL engine as the service layer. However, that might still lead to multiple copies of data and might require further consolidation of data using additional batch jobs in order to present the user with a single consistent and integrated view of data. This issue can be overcome by making use of Delta Lake as a persistent data storage layer for the Lambda Architecture.

Since...