Controlled rollouts and A/B testing

Deployments of ML models can be considered similar to that of features and changes in application development. Consider a retrained and reoptimized model to be similar to a small UI change in the application when rolling a model out to your users. This might not be obvious at first, but put yourself into a user's shoes in a scenario where suddenly a recommendation algorithm changes from its previous behavior.

Rollouts should never be uncontrolled or based on personal feelings or preferences—they should be based solely on hard metrics. The best and most systematic way to roll out new features and updates to your users is to define a key metric, roll out your new model to one section of the users (group B) and serve the old model to the remaining section of the users (group A). Once the metrics for the users in group B exceed the metrics from group A over a defined period of time, you can confidently roll out the feature to all your...