You're reading from Data Engineering with Databricks Cookbook Build effective data and AI solutions using Apache Spark, Databricks, and Delta Lake

Product type Paperback

Published in May 2024

Publisher Packt

ISBN-13 9781837633357

Length 438 pages

Edition 1st Edition

Tools

Apache Spark

Concepts

Data Engineering

Author (1):

Pulkit Chadha

View More author details

Table of Contents (16) Chapters

Preface

1. Part 1 – Working with Apache Spark and Delta Lake FREE CHAPTER

2. Chapter 1: Data Ingestion and Data Extraction with Apache Spark

3. Chapter 2: Data Transformation and Data Manipulation with Apache Spark

4. Chapter 3: Data Management with Delta Lake

5. Chapter 4: Ingesting Streaming Data

6. Chapter 5: Processing Streaming Data

7. Chapter 6: Performance Tuning with Apache Spark

8. Chapter 7: Performance Tuning in Delta Lake

9. Part 2 – Data Engineering Capabilities within Databricks

10. Chapter 8: Orchestration and Scheduling Data Pipeline with Databricks Workflows

11. Chapter 9: Building Data Pipelines with Delta Live Tables

12. Chapter 10: Data Governance with Unity Catalog

13. Chapter 11: Implementing DataOps and DevOps on Databricks

14. Index

Why subscribe?

15. Other Books You May Enjoy

Implementing DataOps and DevOps on Databricks

DataOps and DevOps are two methodologies that aim to improve the quality, speed, and efficiency of data and software development processes. DataOps focuses on the end-to-end orchestration of data pipelines, from data ingestion to analysis and visualization. DevOps focuses on the continuous integration and delivery of software applications, from code development to deployment and monitoring.

Databricks supports both DataOps and DevOps practices by offering various features and tools that enable users to collaborate, automate, and optimize their data and code workflows.

In the chapter, you will learn how to implement DataOps and DevOps on Databricks. We will cover the following recipes:

Using Databricks Repos to store code in Git
Automating tasks by using the Databricks command-line interface (CLI)
Using the Databricks VSCode extension for local development and testing
Using Databricks Asset Bundles (DABs)
Leveraging...