The Data Intelligence Platform allows your entire organization to leverage your data and AI. It’s built on a lakehouse architecture to provide an open, unified foundation for all data and governance layers. It is powered by a Data Intelligence Engine, which understands the context of your data. For practical purposes, let’s talk about the components of the Databricks Data Intelligence Platform:

Figure 1.1 – The components of the Databricks Data Intelligence Platform

Let’s check out the following list with the descriptions of the items in the figure:

• Delta Lake: The data layout within the Data Intelligence Platform is automatically optimized based on common data usage patterns
• Unity Catalog: A unified governance model to secure, manage, and share your data assets
• Data Intelligence Engine: This uses AI to enhance the platform’s capabilities
• Databricks...

Databricks’ implementation of a lakehouse architecture is unique. Databricks’ foundation is built on a Delta-formatted data lake that Unity Catalog governs. Therefore, it combines a data lake’s scalability and cost-effectiveness with a data warehouse’s governance. This means not only are table-level permissions managed through access control lists (ACLs) but file and object-level access are also regulated. This change in architecture from a data lake and/or a data warehouse to a unified platform is ideal – a lakehouse facilitates a wide range of new use cases for analytics, business intelligence, and data science projects across an organization. See the Introduction to Data Lakes blog post in the Further reading section for more information on lakehouse benefits.

This section will discuss the importance of open source frameworks and two critical advantages they provide – transparency and flexibility...

This book is heavily project-based. Each chapter starts with an overview of the important concepts and Data Intelligence Platform features that will prepare you for the main event – the Applying our learning sections. Every Applying our learning section has a Technical requirements section so that you know what technical resources you will need, in addition to your Databricks workspace and GitHub repository, to complete the project work in the respective chapter.

Technical requirements

Here are the technical requirements needed to get started with the hands-on examples used throughout this book:

• We use Kaggle for two of our datasets. If you do not already have an account, you will need to create one.
• Throughout the book, we will refer to code in GitHub. Create an account if you do not already have one.

Getting to know your data

There are four main projects that progress sequentially throughout the book. In each subsequent chapter...

In this chapter, we introduced you to Databricks ML in Action. We emphasized that the Databricks Data Intelligence Platform is designed with openness, flexibility, and tooling freedom in mind, which greatly accelerates productivity. Additionally, we’ve given you a sneak peek at the projects and the associated datasets that will be central to this book.

Now that you’ve gained a foundational understanding of the Data Intelligence Platform, it’s time to take the next step. In the upcoming chapter, we’ll guide you through setting up your environment and provide instructions on downloading the project data. This will prepare you for the practical, hands-on ML experiences that lie ahead in this journey.

Let’s test ourselves on what we’ve learned by going through the following questions:

1. How will you use this book? Do you plan to go cover to cover or pick certain sections out? Have you chosen sections of interest?
2. We covered why transparency in modeling is critical to success. How does Databricks’ glass-box approach to AutoML support this?
3. Databricks has developed a new way of uniting the open data formats, called UniForm. Which data formats does UniForm unite?
4. Delta is the foundation of the lakehouse architecture. What is one of the benefits of using Delta?
5. What is the main advantage of using the Delta file format for large-scale data processing over simple Parquet files in Databricks?

After putting thought into the questions, compare your answers to ours:

1. We cannot answer this question, but we hope you learn something you can use in your career soon!
2. The glass-box approach supports transparency by providing the code run for each run in the experiment and the best run, thus enabling reusability and reproducibility.
3. Apache Iceberg, Apache Hudi, and Linux Foundation Delta Lake (an open source/unmanaged version of Delta).
4. There are several. Here are a few:
   • Open protocol (no vendor lock-in)
   • Speed
   • Change data capture
   • Time travel
5. While Parquet also provides columnar storage and has efficient read/write operations, its lack of ACID transaction capabilities distinguishes Delta Lake.

In this chapter, we introduced vital technologies. Look at these resources to go deeper into the areas that interest you most:

• YouTube video – Introduction to Databricks Data Intelligence Platform: https://youtu.be/E885Ld3N2As?si=1NPg85phVH8RhayO
• Introduction to Data Lakes: https://www.databricks.com/discover/data-lakes
• 5 Steps to a Successful Data Lakehouse by Bill Inmon, father of the data warehouse: https://www.databricks.com/resources/ebook/building-the-data-lakehouse
• Delta Lake: Up & Running by O’Reilly: https://www.databricks.com/resources/ebook/delta-lake-running-oreilly
• Delta Lake: The Definitive Guide: https://www.oreilly.com/library/view/delta-lake-the/9781098151935/
• Comparing Apache Spark and Databricks: https://www.databricks.com/spark/comparing-databricks-to-apache-spark
• Databricks MLflow: https://www.databricks.com/product/managed-mlflow
• Databricks Community Edition FAQ: https://www.databricks.com/product...