Chapter 4: Delta Lake with Azure Databricks
In this chapter, we will learn how to make use of Azure Databricks by showing how easy it is to work with Delta tables, as well as how to process data and integrate different solutions in Azure Databricks. We will begin by introducing Delta Lake and how to ingest data with it using either using partner integrations, the COPY INTO command, and the Azure Databricks Auto Loader. Then, we'll show you how to process the data once it has been loaded, as well as how we can use advance features in order to process and optimize ETLs that rely on streams of data.
We will learn how to store and process data efficiently using Delta by covering the following topics:
- Introducing Delta Lake
- Ingesting data using Delta Lake
- Batching table reads and writes in queries
- Querying past states of a table
- Streaming table read and writes
