Using SQL

After using the previous Scala example to create a data frame from a JSON input file on HDFS, we can now define a temporary table based on the data frame and run SQL against it.

The following example shows you the temporary table called washing_flat being defined and a row count being created using count(*):

The schema for this data was created on the fly (inferred). This is a very nice function of the Apache Spark DataSource API that has been used when reading the JSON file from HDFS using the SparkSession object. However, if you want to specify the schema on your own, you can do so.

Defining schemas manually

So first, we have to import some classes. Follow the code to do this:

import org.apache.spark.sql.types._

So...