Statistical Significance Testing

Often, an analyst is interested in comparing the statistical properties of two groups, or perhaps just one group before and after a change. Of course, the difference between these two groups may just be due to chance.

An example of where this comes up is in marketing A/B tests. Companies often test two different types of landing pages for a product and measure how many clicks it will receive on each of the landing pages. For example, if you make the image of a product two times larger, will this make people more likely to click it? You may find that 10% of the visitors for variation A of the landing page clicked on the product, and 11% for variation B. So, does that mean variation B is 10% better than A or is this just a result of day-to-day variance? You need a method based on statistics to determine just that.

Statistical significance testing is the method of determining whether the data that you have supports a certain hypothesis. To build...