Continuous aggregation with structured streaming
As noted in earlier chapters, the execution of Spark SQL or DataFrame queries revolves around building a logical plan, choosing a physical plan (of the many generated physical plans) based on its cost optimizer, and then generating the code (that is, code gen) via the Spark SQL Engine Catalyst Optimizer. What structured streaming introduces is the concept of an incremental execution plan. That is, structured streaming repeatedly applies the execution plan for every new block of data it receives. This way, the Spark SQL engine can take advantage of the optimizations included within Spark DataFrames and apply them to an incoming data stream. Because structured streaming is built on top of Spark DataFrames, this means it will also be easier to integrate other DataFrame-optimized components, including MLlib, GraphFrames, TensorFrames, and so on:
Getting ready
For these Apache Spark Streaming examples, we will be creating and executing a console...
