Observational studies
Randomization is critical. In online advertising, the standard approach to achieve this is to leave choosing which users enter the test and control groups to the end of the ad delivery pipeline, after the targeting and bidding.
Figure 13.5 – Flow of an ad request for testing split
We have at least three rates we can measure:
Figure 13.6 – Flow of an ad request for testing split 2
We have R c for the control group, R T L for the test group that lost the bid, and R T W for the test group that won the bid. We want to find R C W, that is, the conversion rate for the users who would have won even if they weren’t served with impressions.
We can estimate it with some assumptions. Note that the ratio, y, is the observable ratio of the number of users who won the bid and were served with impressions, over the number of users who lost the bid. By assuming...
