Chapter 9: Delta for Reproducible Machine Learning Pipelines

"Repetition is the mother of learning, the father of action, which makes it the architect of accomplishment."

– Zig Ziglar, American author and motivational speaker

In previous chapters, we established the pivotal nature of Delta in architecting data pipelines. What about Machine Learning (ML) pipelines? They involve different personas with different skills and needs. ML has been around for a while; what has changed lately is broad access to large datasets and affordable compute, which has now made it possible for everyone to tinker with ML. Can Delta stand the litmus test of building a reproducible ML pipeline just as effectively as a data pipeline? There are specific challenges and nuances in building a model, staging it in production, and repeating the process over and over again. In this chapter, we will look into these challenges and map the capabilities of Delta...