Search icon CANCEL
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Conferences
Free Learning
Arrow right icon
Arrow up icon
GO TO TOP
Data Engineering with Apache Spark, Delta Lake, and Lakehouse

You're reading from   Data Engineering with Apache Spark, Delta Lake, and Lakehouse Create scalable pipelines that ingest, curate, and aggregate complex data in a timely and secure way

Arrow left icon
Product type Paperback
Published in Oct 2021
Publisher Packt
ISBN-13 9781801077743
Length 480 pages
Edition 1st Edition
Languages
Arrow right icon
Author (1):
Arrow left icon
Manoj Kukreja Manoj Kukreja
Author Profile Icon Manoj Kukreja
Manoj Kukreja
Arrow right icon
View More author details
Toc

Table of Contents (17) Chapters Close

Preface 1. Section 1: Modern Data Engineering and Tools
2. Chapter 1: The Story of Data Engineering and Analytics FREE CHAPTER 3. Chapter 2: Discovering Storage and Compute Data Lakes 4. Chapter 3: Data Engineering on Microsoft Azure 5. Section 2: Data Pipelines and Stages of Data Engineering
6. Chapter 4: Understanding Data Pipelines 7. Chapter 5: Data Collection Stage – The Bronze Layer 8. Chapter 6: Understanding Delta Lake 9. Chapter 7: Data Curation Stage – The Silver Layer 10. Chapter 8: Data Aggregation Stage – The Gold Layer 11. Section 3: Data Engineering Challenges and Effective Deployment Strategies
12. Chapter 9: Deploying and Monitoring Pipelines in Production 13. Chapter 10: Solving Data Engineering Challenges 14. Chapter 11: Infrastructure Provisioning 15. Chapter 12: Continuous Integration and Deployment (CI/CD) of Data Pipelines 16. Other Books You May Enjoy

Introducing data lakes

Over the last few years, the markers for effective data engineering and data analytics have shifted. Up to now, organizational data has been dispersed over several internal systems (silos), each system performing analytics over its own dataset.

Additionally, it has been difficult to interface with external datasets for extending the spectrum of analytic workloads. As a result, it has been difficult for these organizations to get a unified view of their data and gain global insights.

In a world where organizations are seeking revenue diversification by fine-tuning existing processes and generating organic growth, a globally unified repository of data has become a core necessity. Data lakes solve this need by providing a unified view of data into the hands of users who can use this data to devise innovative techniques for the betterment of mankind.

The following diagram outlines the characteristics of a data lake:

Figure 2.1 – Characteristics of a data lake

Figure 2...

lock icon The rest of the chapter is locked
Register for a free Packt account to unlock a world of extra content!
A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.
Unlock this book and the full library FREE for 7 days
Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of
Renews at €18.99/month. Cancel anytime