Hive integration

Spark is integrated really well with Hive, though it does not include much of its dependencies and expects them to be available in its classpath. The following steps explain how to integrate Spark with Hive:

1. Place hive-site.xml, core-site.xml, and hdfs-site.xml files in the SPARK_HOME/conf folder.
2. Instantiate SparkSession with Hive support and, if hive-site.xml is not configured, then the context automatically creates metastore_db in the current directory and creates a warehouse directory configured by spark.sql.warehouse.dir, which defaults to the directory spark-warehouse.

SparkSession sparkSession = SparkSession
  .builder()
  .master("local")
  .config("spark.sql.warehouse.dir","Path of Warehouse")
  .appName("DatasetOperations")
  .enableHiveSupport()
  .getOrCreate();

3. Once we have created a SparkSession with Hive support enabled, we can proceed to use it with the added benefits of query support from Hive. One way to identify the difference between Hive query function support...