Bringing it all together

So far, we have covered the essential elements of batch and stream ingestion and processing. Now, let's bring these two types of processing together to define the Lambda architecture pattern.

Figure 3.12 – Lambda architecture pattern

The preceding diagram depicts a Lambda architecture pattern. A Lambda architecture pattern has three layers: the batch layer, the speed layer, and the serving layer.

The batch layer

The following diagram illustrates batch layer processing in a Lambda architecture:

Figure 3.13 – The batch layer in a Lambda architecture

Batch layer processing consists of ingesting the data into the raw data store of the data lake using pull or push methodologies through a batch data ingestion service. Once the data has been ingested in the raw data store, a batch processing service is initiated. The batch processing service employs a distributed computing engine for faster...