You're reading from Data Engineering with Databricks Cookbook Build effective data and AI solutions using Apache Spark, Databricks, and Delta Lake

Product type Paperback

Published in May 2024

Publisher Packt

ISBN-13 9781837633357

Length 438 pages

Edition 1st Edition

Tools

Apache Spark

Concepts

Data Engineering

Author (1):

Pulkit Chadha

View More author details

Table of Contents (16) Chapters

Preface

1. Part 1 – Working with Apache Spark and Delta Lake FREE CHAPTER

2. Chapter 1: Data Ingestion and Data Extraction with Apache Spark

3. Chapter 2: Data Transformation and Data Manipulation with Apache Spark

4. Chapter 3: Data Management with Delta Lake

5. Chapter 4: Ingesting Streaming Data

6. Chapter 5: Processing Streaming Data

7. Chapter 6: Performance Tuning with Apache Spark

8. Chapter 7: Performance Tuning in Delta Lake

9. Part 2 – Data Engineering Capabilities within Databricks

10. Chapter 8: Orchestration and Scheduling Data Pipeline with Databricks Workflows

11. Chapter 9: Building Data Pipelines with Delta Live Tables

12. Chapter 10: Data Governance with Unity Catalog

13. Chapter 11: Implementing DataOps and DevOps on Databricks

14. Index

Why subscribe?

15. Other Books You May Enjoy

Triggering jobs based on file arrival

File arrival triggers can be used to start your Databricks Workflow when new files show up in an external location such as Amazon S3 or Azure storage. This feature is useful when new data does not come regularly and a scheduled job would not work well. Every minute, file arrival triggers look for new files. They do not have any extra cost except for the cloud provider costs for listing files in the storage location.

In this recipe, you will learn how to set up file arrival triggers to run Databricks Workflows.

Getting ready

Before you can use file arrival triggers, you need to meet the following conditions:

Your workspace should have Unity Catalog activated. Unity Catalog is a unified governance solution for data and AI assets on the lakehouse.
An external location that belongs to the Unity Catalog metastore is recommended. This is an object that has both a cloud storage path and a storage credential that allows access to that...