You're reading from Azure Data Engineering Cookbook Get well versed in various data engineering techniques in Azure using this recipe-based guide

Product type Paperback

Published in Sep 2022

Publisher Packt

ISBN-13 9781803246789

Length 608 pages

Edition 2nd Edition

Languages

SQL

Tools

Azure

Concepts

Data Engineering

Authors (3):

Ahmad Osama

Nagaraj Venkatesan

Luca Zanna

View More author details

Table of Contents (16) Chapters

Preface

1. Chapter 1: Creating and Managing Data in Azure Data Lake

2. Chapter 2: Securing and Monitoring Data in Azure Data Lake FREE CHAPTER

3. Chapter 3: Building Data Ingestion Pipelines Using Azure Data Factory

4. Chapter 4: Azure Data Factory Integration Runtime

5. Chapter 5: Configuring and Securing Azure SQL Database

6. Chapter 6: Implementing High Availability and Monitoring in Azure SQL Database

7. Chapter 7: Processing Data Using Azure Databricks

8. Chapter 8: Processing Data Using Azure Synapse Analytics

9. Chapter 9: Transforming Data Using Azure Synapse Dataflows

10. Chapter 10: Building the Serving Layer in Azure Synapse SQL Pool

11. Chapter 11: Monitoring Synapse SQL and Spark Pools

12. Chapter 12: Optimizing and Maintaining Synapse SQL and Spark Pools

13. Chapter 13: Monitoring and Maintaining Azure Data Engineering Pipelines

14. Index

Why subscribe?

15. Other Books You May Enjoy

Analyzing a query plan and fixing table distribution

Synapse dedicated SQL Pool is a distributed database engine that is comprised of one control node and one or more compute nodes. A control node is like the brain of the database engine and all the queries that are submitted by client applications are received by the control node. The control node splits the work that needs to be done to process the query and assigns it to compute nodes in Synapse dedicated SQL Pool. The compute nodes process the work assigned to them and return the partial results to the control node. The control node combines the results from compute nodes, performs any additional processing, and returns the final output to the client application. The number of compute nodes depends on the data warehouse units (DWUs) of the Synapse dedicated SQL pool, with SQL pools with higher DWUs getting more compute nodes. For example, a DWU1500 SQL pool has three compute nodes, while a DWU30000 SQL pool has 60 compute nodes...