Chapter 4: Unifying Batch and Streaming with Delta

"We are only as strong as we are united, as weak as we are divided".

– J.K. Rowling, author of the Harry Potter series

In the last chapter, we examined Delta's capabilities and how it solves the challenges of traditional data lakes to give you curated data that is the foundation for sound insights without having to solve common operational problems over and over again. In this chapter, we will look at the two patterns of ingestion in data systems, namely, batch and streaming. Traditionally, they would have required two separate pipelines and the associated cost and effort to create, maintain, and reconcile data between the two pipelines. Thanks to protocols such as Delta, these two pipelines can now be consolidated.

In particular, we will be covering the following topics:

• Moving toward real-time systems
• Streaming ETL
• Handling streaming scenarios...