Iterating on models through A/B testing

In the examples above and in the previous chapters of this volume, we have primarily examined analytical systems in terms of their predictive ability. However, these measures do not necessarily ultimately quantify the kinds of outcomes that are meaningful to the business, such as revenue and user engagement. In some cases, this shortcoming is overcome by converting the performance statistics of a model into other units that are more readily understood for a business application. For example, in our preceding churn model, we might multiply our prediction of 'cancel' or 'not-cancel' to generate a predicted dollar amount lost through subscriber cancellation.

In other scenarios, we are fundamentally unable to measure a business outcome using historical data. For example, in trying to optimize a search model, we can measure whether a user clicked a recommendation and whether they ended up purchasing anything after clicking. Through such retrospective analysis...