Streaming table read and writes

Although we have already mentioned that structured streaming is a core part of the Apache Spark API, in this section, we will dive into how we can use it as a reliable stream processing engine, in which computation can be performed in the same way as batch computation is performed on static data. Along with Auto Loader, it will automatically handle data being streamed into Delta tables without the common inconveniences such as merging small files produced by low latency ingestion, running concurrent batch jobs when working with several streams, and, as we discussed earlier, keeping track of the files available for being streamed into tables.

Let's learn how to stream data into Delta tables in Azure Databricks.

Streaming from Delta tables

You can use a Delta table as a stream source for streaming queries. The query will then process any existing data in the table and any incoming data from the stream. Let's take a look:

• We can...