Productionizing ML on Databricks

“Production is 80% of the work.” — Matei Zaharia

Once you’ve refined your model and have satisfactory results, you are ready to put it into production. We’ve now entered the field of Machine learning operations (MLOps)! Unfortunately, this is where many data scientists and ML engineers get stuck, and it’s not uncommon for companies to struggle here. Implementing models in production is much more complex than running models ad hoc because MLOps requires distinct tools and skill sets and sometimes entirely new teams. MLOps is an essential part of the data science process because the actual value of a model is often only realized post-deployment.

You can think of MLOps as combining DevOps, DataOps, and ModelOps. MLOps is often divided into two parts: inner and outer loops. The inner loop covers the data science work and includes tracking various stages of the model development and experimentation process...