Spark SQL

Spark SQL is a Spark module for structured data processing that was introduced in Spark 1.0. This module is a tightly integrated relational engine that inert-operates with the core Spark API. It enables data engineers to write applications that load structured data from disparate sources and join them as a unified, and possibly continuous, Excel-like data frames; and then they can implement complex ETL workflows and advanced analytics.

The Spark 2.0 release brought in significant unification of APIs and expanded the SQL capabilities, including support for subqueries. The Dataset API and DataFrames API are now unified, with DataFrames being a "kind" of Datasets. The unified APIs build the foundation for Spark's future, spanning across all libraries. Developers can impose "structure" onto their data and can work with high-level declarative APIs, thereby improving performance as well as their productivity. The performance gains come as a result of the underlying optimization layer...