Processing Data Using Spark Structured Streaming

Structured Streaming is a feature in Apache Spark where the incoming stream is treated as an unbounded table. The incoming streaming data is continuously appended to the table. This feature makes it easy to write streaming queries, as you can now write streaming transformations in the same way you handle table-based transformations. Hence, the same Spark batch-processing syntax can be applied here, too.

Note

This section primarily focuses on the Process data by using Spark structured streaming concept of the DP-203: Data Engineering on Microsoft Azure exam.

Spark treats the Structured Streaming queries as incremental queries on an unbounded table and runs them at frequent intervals to continuously process the data. Spark supports three writing modes for the output of Structured Streaming:

• Complete mode: In this mode, the entire output (also known as the result table) is written to the sink. The sink could be a Blob Storage...