Feature Engineering on Databricks

“Applied machine learning is basically feature engineering.”

– Andrew Ng

As we progress from Chapter 4, where we harnessed the power of Databricks to explore and refine our datasets, we are now ready to delve into the components of Databricks that enable the next step: feature engineering. We will start by covering Databricks Feature Engineering (DFE) in Unity Catalog to show you how you can efficiently manage engineered features using Unity Catalog (UC). Understanding how to leverage DFE in UC is crucial for creating reusable and consistent features across training and inference. Next, you will learn how to leverage Sparka Structured Streaming for calculating features on a stream, which allows you to create stateful features needed for models to perform quick decision-making. Feature engineering is a broad topic. We will focus on how the DI Platform facilitates the development of certain feature categories, such as point...