Chapter 2: Creating Robust Data Pipelines and Data Transformation
In this chapter, we will cover how to load and enrich data using the power of Apache Spark in Azure Synapse Analytics. We will learn about and understand various concepts and recipes for writing Spark data frames to read data from Azure Data Lake Storage (ADLS) and writing to a SQL pool using PySpark.
This chapter comprises the following recipes:
- Reading and writing data from ADLS Gen2 using PySpark
- Visualizing data in a Synapse notebook