You're reading from Distributed Data Systems with Azure Databricks Create, deploy, and manage enterprise data pipelines

Product type Paperback

Published in May 2021

Publisher Packt

ISBN-13 9781838647216

Length 414 pages

Edition 1st Edition

Languages

Python

Tools

Azure

Concepts

Data Science

Author (1):

Alan Bernardo Palacio

View More author details

Table of Contents (17) Chapters

Preface

1. Section 1: Introducing Databricks

2. Chapter 1: Introduction to Azure Databricks FREE CHAPTER

3. Chapter 2: Creating an Azure Databricks Workspace

4. Section 2: Data Pipelines with Databricks

5. Chapter 3: Creating ETL Operations with Azure Databricks

6. Chapter 4: Delta Lake with Azure Databricks

7. Chapter 5: Introducing Delta Engine

8. Chapter 6: Introducing Structured Streaming

9. Section 3: Machine and Deep Learning with Databricks

10. Chapter 7: Using Python Libraries in Azure Databricks

11. Chapter 8: Databricks Runtime for Machine Learning

12. Chapter 9: Databricks Runtime for Deep Learning

13. Chapter 10: Model Tracking and Tuning in Azure Databricks

14. Chapter 11: Managing and Serving Models with MLflow and MLeap

15. Chapter 12: Distributed Deep Learning in Azure Databricks

16. Other Books You May Enjoy

Ingesting data using Delta Lake

Data can be ingested into Delta Lake in several ways. Azure Databricks offers several integrations with Partners, which provide data sources that are loaded as Delta tables. We can copy a file directly into a table, use AutoLoader, or create a new streaming table. Let's take a deeper look at this.

Partner integrations

Azure Databricks allows us to connect to different partners that provide data sources. These are easy to implement and provide scalable ingestion.

We can view the options that we have for ingesting data from Partner Integrations when creating a new table in the UI, as shown in the following screenshot:

Figure 4.1 – Ingesting data from Partner Integrations

Some of these integrations, such as Qlink, allow you to get data from multiple data sources such as Oracle, Microsoft SQL Server, and SAP and load them into Delta Lake. Of course, these integrations require you to have a subscription to...