Shredding JSON
Shredding refers to the process of extracting data from JSON files into tables. Spark, Synapse SQL pools, and ADF provide native support to extract data from JSON. Let's look at examples for each of the services.
Extracting values from JSON using Spark
Spark can directly read JSON files and extract the schema from them. Here is a simple code snippet that can accomplish the JSON read:
val dfJSON = spark.read.json("abfss://path/to/json/*.json")
dfJSON.printSchema()
dfJSON.show(false)
Here is how the output looks:
Figure 8.24 – Output of JSON operation
You can also manually specify the schema, as shown in the following example:
val driverSchema = new StructType()
.add("firstname", StringType)
.add("middlename", StringType)
. . .
.add("salary",IntegerType)
val dfJSON = spark.read.schema(driverSchema).json("abfss...
