You're reading from Data Engineering with Databricks Cookbook Build effective data and AI solutions using Apache Spark, Databricks, and Delta Lake

Product type Paperback

Published in May 2024

Publisher Packt

ISBN-13 9781837633357

Length 438 pages

Edition 1st Edition

Tools

Apache Spark

Concepts

Data Engineering

Author (1):

Pulkit Chadha

View More author details

Table of Contents (16) Chapters

Preface

1. Part 1 – Working with Apache Spark and Delta Lake FREE CHAPTER

2. Chapter 1: Data Ingestion and Data Extraction with Apache Spark

3. Chapter 2: Data Transformation and Data Manipulation with Apache Spark

4. Chapter 3: Data Management with Delta Lake

5. Chapter 4: Ingesting Streaming Data

6. Chapter 5: Processing Streaming Data

7. Chapter 6: Performance Tuning with Apache Spark

8. Chapter 7: Performance Tuning in Delta Lake

9. Part 2 – Data Engineering Capabilities within Databricks

10. Chapter 8: Orchestration and Scheduling Data Pipeline with Databricks Workflows

11. Chapter 9: Building Data Pipelines with Delta Live Tables

12. Chapter 10: Data Governance with Unity Catalog

13. Chapter 11: Implementing DataOps and DevOps on Databricks

14. Index

Why subscribe?

15. Other Books You May Enjoy

Quarantining bad data with Delta Live Tables in Databricks

Quarantining bad data means sending it to a separate location for further inspection and correction. You can also replay the quarantined data once you have fixed the issues. With Delta Live Tables, you can quarantine bad data that does not meet your expectations or requirements. You can define expectations for your data using SQL or Python expressions, and specify how to handle records that fail those expectations. You can choose to fail, drop, alert, or quarantine the bad data.

In this recipe, you will learn how to use Delta Live Tables to quarantine bad data in your data pipelines, and how to backfill the quarantined data after resolving the problems.

How to do it…

Incremental ingestion with an autoloader: Create a dataset from the farmers_markets_geographic_data CSV data by creating a streaming table. We are defining a streaming table to denote that this is an incremental append-only load from the CSV...