You're reading from Data Engineering with Databricks Cookbook Build effective data and AI solutions using Apache Spark, Databricks, and Delta Lake

Product type Paperback

Published in May 2024

Publisher Packt

ISBN-13 9781837633357

Length 438 pages

Edition 1st Edition

Tools

Apache Spark

Concepts

Data Engineering

Author (1):

Pulkit Chadha

View More author details

Table of Contents (16) Chapters

Preface

1. Part 1 – Working with Apache Spark and Delta Lake FREE CHAPTER

2. Chapter 1: Data Ingestion and Data Extraction with Apache Spark

3. Chapter 2: Data Transformation and Data Manipulation with Apache Spark

4. Chapter 3: Data Management with Delta Lake

5. Chapter 4: Ingesting Streaming Data

6. Chapter 5: Processing Streaming Data

7. Chapter 6: Performance Tuning with Apache Spark

8. Chapter 7: Performance Tuning in Delta Lake

9. Part 2 – Data Engineering Capabilities within Databricks

10. Chapter 8: Orchestration and Scheduling Data Pipeline with Databricks Workflows

11. Chapter 9: Building Data Pipelines with Delta Live Tables

12. Chapter 10: Data Governance with Unity Catalog

13. Chapter 11: Implementing DataOps and DevOps on Databricks

14. Index

Why subscribe?

15. Other Books You May Enjoy

Working with nested data structures in Apache Spark

In this recipe, we will walk you through the step-by-step process of handling nested data structures such as arrays, maps, and so on with Apache Spark. This recipe will equip you with the essential knowledge and practical skills needed to work with complex data types using Apache Spark’s distributed computing capabilities.

How to do it…

Import libraries: Import the required libraries and create a SparkSession object: SparkSession is a unified entry point for Spark applications. It provides a simplified way to interact with various Spark functionalities, such as resilient distributed datasets (RDDs), DataFrames, datasets, SQL queries, streaming, and more. You can create a SparkSession object using the builder method, which allows you to configure the application name, master URL, and other options. We will also define SparkContext, which is the entry point to any Spark functionality. It represents the connection...