Experimentation

For a randomized experiment, you split the users into two groups (test and control), and then you apply the treatment to the test group, and you measure the conversion rate of both groups. Random selection of who is in each group is essential and ensures that the groups are comparable and that you are controlling for unobserved factors.

To find the credible interval of lift, we can do the following:

Pr(a < L < b) = ∫ ∫ a<L<b L(R t, R c)Pr(R t, R c)d R t d R c

If the random experiment is done correctly, then the distribution of R t and R c is independent, and we can write the following:

Pr(R t, R c) = Pr(R t| k t, n t)Pr(R c| k c, n c)

Now, we can simulate. We know that rates R c and R t will follow a beta distribution, so we can simulate a large number of values for R c and R t, and then calculate the lift for each pair of...