Applying our learning
You will ingest the data before diving into each project’s core “data science” aspects. We’ve discussed how Auto Loader, schema evolution, and DLT can format your data in storage. You’ll notice that the following projects use different patterns to load data into the Bronze layer. The streaming transactions project uses Auto Loader to ingest incoming JSON files. You will transform the data with DLT and Structure Streaming independently, allowing you to get experience with both methods.
Technical requirements
Before we begin, review the technical requirements needed to complete the hands-on work in this chapter.
Databricks ML Runtime includes several pre-installed libraries useful for ML and data science projects. For this reason, we will use clusters with an ML runtime.
Project – streaming transactions
The next step in our streaming transactions project is to build out the Bronze layer, or the raw data we...
