You're reading from Data Engineering with Databricks Cookbook Build effective data and AI solutions using Apache Spark, Databricks, and Delta Lake

Product type Paperback

Published in May 2024

Publisher Packt

ISBN-13 9781837633357

Length 438 pages

Edition 1st Edition

Tools

Apache Spark

Concepts

Data Engineering

Author (1):

Pulkit Chadha

View More author details

Table of Contents (16) Chapters

Preface

1. Part 1 – Working with Apache Spark and Delta Lake FREE CHAPTER

2. Chapter 1: Data Ingestion and Data Extraction with Apache Spark

3. Chapter 2: Data Transformation and Data Manipulation with Apache Spark

4. Chapter 3: Data Management with Delta Lake

5. Chapter 4: Ingesting Streaming Data

6. Chapter 5: Processing Streaming Data

7. Chapter 6: Performance Tuning with Apache Spark

8. Chapter 7: Performance Tuning in Delta Lake

9. Part 2 – Data Engineering Capabilities within Databricks

10. Chapter 8: Orchestration and Scheduling Data Pipeline with Databricks Workflows

11. Chapter 9: Building Data Pipelines with Delta Live Tables

12. Chapter 10: Data Governance with Unity Catalog

13. Chapter 11: Implementing DataOps and DevOps on Databricks

14. Index

Why subscribe?

15. Other Books You May Enjoy

Processing Streaming Data

Streaming data is data that is continuously generated and updated in real time, such as sensor readings, weblogs, social media posts, online transactions, and more. Streaming data can provide valuable insights into the current state and trends of various domains, such as e-commerce, finance, health care, gaming, and the Internet of Things (IoT). However, streaming data also poses many challenges for data ingestion and processing, such as scalability, reliability, fault tolerance, latency, and consistency.

Apache Spark is a popular open source framework for large-scale distributed data processing. Apache Spark Structured Streaming is an extension of Spark SQL that enables scalable and fault-tolerant processing of streaming data using a declarative API based on DataFrames and datasets. Apache Spark Structured Streaming supports various sources and sinks for streaming data, such as Kafka, Flume, Hadoop Distributed File System (HDFS), Amazon Simple Storage...