Introduction to Spark SQL

Spark SQL brings native support for SQL to Apache Spark and unifies the process of querying data stored both in Spark DataFrames and in external data sources. Spark SQL unifies DataFrames and relational tables and makes it easy for developers to intermix SQL commands with querying external data for complex analytics. With the release of Apache Spark 1.3, Spark DataFrames powered by Spark SQL became the de facto abstraction of Spark for expressing data processing code, while resilient distributed datasets (RDDs) still remain Spark's core abstraction method, as shown in the following diagram:

Figure 12.2 – Spark SQL architecture

As shown in the previous diagram, you can see that most of Spark's components now leverage Spark SQL and DataFrames. Spark SQL provides more information about the structure of the data and the computation being performed, and the Spark SQL engine uses this extra information to perform additional...