You're reading from Azure Data Engineering Cookbook Get well versed in various data engineering techniques in Azure using this recipe-based guide

Product type Paperback

Published in Sep 2022

Publisher Packt

ISBN-13 9781803246789

Length 608 pages

Edition 2nd Edition

Languages

SQL

Tools

Azure

Concepts

Data Engineering

Authors (3):

Ahmad Osama

Nagaraj Venkatesan

Luca Zanna

View More author details

Table of Contents (16) Chapters

Preface

1. Chapter 1: Creating and Managing Data in Azure Data Lake

2. Chapter 2: Securing and Monitoring Data in Azure Data Lake FREE CHAPTER

3. Chapter 3: Building Data Ingestion Pipelines Using Azure Data Factory

4. Chapter 4: Azure Data Factory Integration Runtime

5. Chapter 5: Configuring and Securing Azure SQL Database

6. Chapter 6: Implementing High Availability and Monitoring in Azure SQL Database

7. Chapter 7: Processing Data Using Azure Databricks

8. Chapter 8: Processing Data Using Azure Synapse Analytics

9. Chapter 9: Transforming Data Using Azure Synapse Dataflows

10. Chapter 10: Building the Serving Layer in Azure Synapse SQL Pool

11. Chapter 11: Monitoring Synapse SQL and Spark Pools

12. Chapter 12: Optimizing and Maintaining Synapse SQL and Spark Pools

13. Chapter 13: Monitoring and Maintaining Azure Data Engineering Pipelines

14. Index

Why subscribe?

15. Other Books You May Enjoy

Optimizing and Maintaining Synapse SQL and Spark Pools

In the previous chapter, we learned how to monitor Synapse dedicated SQL and Spark pools. As a natural follow-up to that, in this chapter, we will cover techniques that can optimize the performance of Synapse SQL and Spark pools. In addition to optimization techniques, we will also share maintenance techniques such as pausing Synapse dedicated SQL pools and configuring a longer backup retention duration for Synapse dedicated SQL pools.

By the end of this chapter, you will have learned how to optimize Synapse dedicated SQL pool by reading a query plan and fixing its distribution, building a replication cache, and configuring result set caching. You will also know how to improve Spark pool performance using Z-ordering and partitioning techniques. In addition to optimization techniques, you will have learned about various maintenance techniques to automatically pause Synapse dedicated SQL pool during quiet periods, configure a...