A/B testing
A/B testing is the practice of testing one model (model A) against another model (model B) to see which one performs better. While the term could technically apply to testing and comparing any models, the usual scenario is to test a new version of a model to improve model performance concerning the business objective.
Vertex AI allows us to deploy more than one model to a single endpoint, as well as control the amount of traffic that is served by each model by using the traffic_split variable, as shown in Figure 10.3:
Figure 10.3: A/B configuration using Vertex AI’s traffic_split
As you will see in the practical exercises in this chapter, if we don’t set any value for the traffic_split variable, the default behavior is to keep all traffic directed to the original model that was already deployed to our endpoint. This is a safety mechanism that prevents unexpected behavior in terms of how our models serve traffic from our clients...
