You're reading from Distributed Data Systems with Azure Databricks Create, deploy, and manage enterprise data pipelines

Product type Paperback

Published in May 2021

Publisher Packt

ISBN-13 9781838647216

Length 414 pages

Edition 1st Edition

Languages

Python

Tools

Azure

Concepts

Data Science

Author (1):

Alan Bernardo Palacio

View More author details

Table of Contents (17) Chapters

Preface

1. Section 1: Introducing Databricks

2. Chapter 1: Introduction to Azure Databricks FREE CHAPTER

3. Chapter 2: Creating an Azure Databricks Workspace

4. Section 2: Data Pipelines with Databricks

5. Chapter 3: Creating ETL Operations with Azure Databricks

6. Chapter 4: Delta Lake with Azure Databricks

7. Chapter 5: Introducing Delta Engine

8. Chapter 6: Introducing Structured Streaming

9. Section 3: Machine and Deep Learning with Databricks

10. Chapter 7: Using Python Libraries in Azure Databricks

11. Chapter 8: Databricks Runtime for Machine Learning

12. Chapter 9: Databricks Runtime for Deep Learning

13. Chapter 10: Model Tracking and Tuning in Azure Databricks

14. Chapter 11: Managing and Serving Models with MLflow and MLeap

15. Chapter 12: Distributed Deep Learning in Azure Databricks

16. Other Books You May Enjoy

PySpark API

We have been using the PySpark API across all sections when describing the features of Azure Databricks without discussing too much of its functionalities and how we can leverage them to make reliable ETL operations when working with big data. PySpark is the Python API for Apache Spark, a cluster-computing framework that is the heart of Azure Databricks.

Main functionalities of PySpark

PySpark allows you to harness the power of distributed computing with the ease of use of Python and it's the default way in which we express our computations through this book unless stated otherwise.

The fundamentals of PySpark lies in the functionality of its sub-packages of which the most central are the following:

PySpark DataFrames: Data stored in rows following a set of named columns. These DataFrames are immutable and allow us to perform lazy computations.
The PySpark SQL module: A higher-abstraction module for processing structured and semi-structured datasets...