Summary
With that, you have come to the end of this interesting chapter. There were lots of examples and screenshots to help you learn the concepts. It might be overwhelming at times, but the easiest way to follow is to open a live Spark, SQL, or ADF session and try to execute the examples in parallel.
You covered a lot of details in this chapter, such as performing transformations in Spark, SQL, and ADF, data cleansing techniques, reading, and parsing JSON data, encoding and decoding, error handling during transformations, normalizing and denormalizing datasets, and finally, a bunch of data exploration techniques. This is one of the important chapters in the syllabus. You should now be able to comfortably build data pipelines with transformations involving Spark, SQL, and ADF.
In the upcoming chapter, you will create resilient batch-processing solutions leveraging Azure’s analytics services including Data Lake Storage, Databricks, Synapse Analytics, and Data Factory.
...
