Building Out Our Bronze Layer

“Data is a precious thing and will last longer than the systems themselves.”

– Tim Berners-Lee, generally credited as the inventor of the World Wide Web

In this chapter, you’ll embark on the beginning of your data journey in the Databricks platform, exploring the fundamentals of the Bronze layer. We recommend employing the Medallion design pattern within the lake house architecture (as described in Chapter 2) to organize your data. We’ll start with Auto Loader, which you can implement with or without Delta Live Tables (DLT) to insert and transform data in your architecture. The benefits of using Auto Loader include quickly transforming new data into the Delta format and enforcing or evolving schemas, which are essential for maintaining consistent data delivery to the business and customers. As a data scientist, strive for efficiency in building your data pipelines and ensuring your data is ready for the steps in...