Ingesting and processing batch data
Let's start by looking at the logical architecture of a data lakehouse:
Figure 3.1 – Data lakehouse logical architecture
The preceding diagram depicts the seven logical layers. Data from the data providers needs to be ingested and transformed. Traditionally, there are two types of batch data ingestion and transformation patterns:
- ETL
- ELT
Understanding these patterns is vital if you wish to understand how they can be combined for batch ingestion and processing in a data lakehouse.
Let's discuss these patterns in detail.
Differences between the ETL and ELT patterns
Let's discuss the differences between these patterns in detail. On the surface, these patterns may seem similar. However, there are differences in their philosophy and the services that are employed to transform data.
ETL
The first pattern is ETL. The following diagram depicts a typical ETL pattern:
