Best practices for updating H2O models

As the famous British statistician George Box stated, All models are wrong, but some are useful. Good modelers are aware of the purpose as well as the limitations of their models. This should be especially true for those who build enterprise models that go into production.

One such limitation is that predictive models as a rule degrade over time. This is largely because, in the real world, things change. Perhaps what we are modeling—customer behavior, for example—itself changes, and the data we collect reflects that change. Even if customer behavior is static but our mix of business changes (think more teenagers and fewer retirees), our model's predictions will likely degrade. In both cases but for different reasons, the population that was sampled to create our predictive model is not the same now as it was before.

Detecting model degradation and searching for its root cause is the subject of diagnostics and model monitoring...