Reproducibility

Now, you know what an ML life cycle would look like and how the platform assists you in every step of your journey. As an individual, you may be able to write every step of the data pipelines and model training and tuning in a single notebook. However, this may cause a problem in teams where different people are working on different parts of the life cycle. Let's say someone wants to run the model training part but the entire process is tied up with one another. Your team may not be able to scale with this approach.

A better and more scalable approach is to write different notebooks for various stages (such as data processing and model training) in your project life cycle and use a workflow engine to tie them up. Using the Kubernetes platform, all the stages will be executed using containers and provide a consistent environment for your project between different runs. The platform provides Airflow, an engine that could be used for creating and executing workflows...