Introducing Structured Streaming

With Spark 2.0, the Apache Spark community is working on simplifying streaming by introducing the concept of structured streaming which bridges the concepts of streaming with Datasets/DataFrames (as noted in the following diagram):

As noted in earlier chapters on DataFrames, the execution of SQL and/or DataFrame queries within the Spark SQL Engine (and Catalyst Optimizer) revolves around building a logical plan, building numerous physical plans, the engine choosing the correct physical plan based on its cost optimizer, and then generating the code (i.e. code gen) that will deliver the results in a performant manner. What Structured Streaming introduces is the concept of an Incremental Execution Plan. When working with blocks of data, structured streaming repeatedly applies the execution plan for every new set of blocks it receives. By running in this manner, the engine can take advantage of the optimizations included within Spark DataFrames/Datasets and apply...