What is a hypothesis test?

A hypothesis test does exactly what it says it does. It tests a hypothesis. Typically, the hypothesis we test is well formulated and can be simply and precisely expressed mathematically. For example, in a scientific experiment, we might want to know whether the average weight of one group of animals is different from that of another group of animals. The hypothesis here can be precisely expressed mathematically as,

Eq. 1

Alternatively, we might be interested in knowing whether website visitors complete purchases at the same rate on variant A of an e-commerce site compared to variant B of the website. In this case, the hypothesis can be expressed as,

Eq. 2

Note that in both these hypotheses, the mathematical expression is in terms of the expectation. In other words, we want to test the hypothesis at the level of populations, not for a finite sample of the A and B populations. In the e-commerce example, we want to know that if everybody...