The t-test is a basic tool in statistics used to compare the means of two samples. Its most strict version assumes that both samples are distributed according to a Gaussian distribution and have the same (unknown) variance.

In statistics, we refer to the null hypothesis as the hypothesis we want to test, and the alternative hypothesis as the one we use when the null is rejected. They are usually referred to as H₀ and H₁.

The t-test tests the equality of means from two populations, and it can be formulated in three possible ways:

• H₀: The means of the two populations are the same versus H₁: in that the means of the populations are different.
• H₀: The mean of population 1 is greater or equal than the one from population 2 versus H₁: the mean of population 1 is smaller than population 2.
• H₀: The mean of population 1 is smaller or equal than the one from population...

The original t-test is designed for two Gaussian samples with equal unknown variance. When the variances are not the same, the degrees of freedom for the test are not the usual ones (the equality of variances is known as homocedasticity). Consequently, we can't calculate the p-values and, by extension, we can't test our hypothesis. This is known as the Fisher-Behrens problem.

It has been found that the t-test (with its usual degrees of freedom) can still be used with moderate departures from the homocedasticity (equality of variances) assumption. But this does not take us very far: translating the idea that the test is robust to departures from this assumption is difficult to operationalize (the impact depends on the sample sizes, the relative differences in the variances, and so on).

If the sample is large enough, we can ignore the problem...

There is a variant of the t-test that can be used when the data is paired (this usually happens when we have two observations for each subject). For example, this may occur if we use a specific program and we want to evaluate its effectiveness by taking one measurement before and after the program is executed. The advantage of this is that the difference (after-before) truly represents the impact of the program that we are evaluating (we are making sure that any possible external variable has been filtered out).

This is much better than having just two samples, where one is taken before the policy was executed and another is one (with different individuals) taken after. If there are differences between those two samples, they might bring trouble to our test. For example, let's suppose that sample1 contains individuals that perform better at work, while sample2...

We could extend our initial example (a website with different color palettes) to something slightly more complex: instead of having just one factor (color), we could have another one (actually, we could have even more than two). For example, we could add the font type that was used on the website and study how those two factors (color and font type) impact the number of purchases that are made. Unfortunately, this adds an extra complication, because one effect might depend on the levels for the other one: for example, the font type might be relevant to explain the number of purchases, but only when the color is red.

The effects for the color and website are usually referred to as main effects, and the interaction between them is referred to as the interaction effect. Before analyzing the main effects, we should always study the interaction effect first: if it is...

In a one-way ANOVA, the sum of squares can be obtained in a straightforward manner. However, in a two-way ANOVA, things get much more complicated because we have at least three possibilities for computing them. For the following examples, let's assume that we have two factors (A and B), each one with their respective levels.

The first possibility is the Type I sum of squares, which can be computed by first calculating the sum of squares for factor A, then the sum of squares of B conditional on A, and finally the sum of squares of the interaction (AB) after including the factors, A and B. This is why this is called the sequential sum of squares. Consequently, the order...

So far, we have explored models where we have different fixed levels for each effect. This makes a lot of sense when we have a set of possible levels for an effect that we control and are interested in measuring. It also makes sense when we have a blocking effect that has a finite (and small) set values (for example, the sex or occupation of a person). In some cases, we will have a huge amount of levels that will be generally unimportant, for example, if we want to measure whether a drug is effective, and we are dealing with multiple observations per person, we want to add a blocking effect for a person. In these cases, we are not interested in the effect per se, but we certainly want to use it as a control variable for our model. A model that uses proper blocks, will be more efficient: think of ANOVA as a method of attributing variability to factors. If we have...

Repeated measures designs contain several measurements for the same experimental unit. This generally occurs when we assign different doses of a treatment to a unit and want to evaluate the response through time.

We now have a fixed effect for time, which is typically coded as 1,2,3..,k that needs to be added. We would expect that this variable/effect will have an impact on the response.

These designs raise four major points, and we need to decide how to solve them:

• Do we think that time has the same impact for all of the experimental units? We have two options for this:
  • If yes, we will just add the time variable as a fixed effect.
  • If no, we will still add the time variable as a fixed effect, plus a random effect for time by subject. What this means is that the time effect can be obtained as the sum of the fixed effect (average for all units) plus the random...

So far, we have worked with univariate data (one variable measured across two samples), and we wanted to test whether the means are equal or not. In certain cases, we might work with multivariate data (for example, measurements of height and weight for certain individuals), and we will be interested in testing the multivariate hypothesis, which is that the means for all of the variables are equal between two groups or not. This is usually formulated as follows:

The difference is that each element is a vector, and we are testing whether all of the elements in a vector are the same between groups. The main assumption here (similar to the univariate t-test) is that the data comes from a multivariate Gaussian distribution.

A relevant question at this stage is whether we can ignore the multi-dimensionality of the problem, and just do univariate t-tests. This would...

We have already explained that ANOVA is the generalization of the t-test for multiple samples. On the other hand, the t-test is designed to work with just one variable, and in case we have multiple variables, we need to use Hotelling T2. Is it possible to extend ANOVA to work with multiple variables? The answer is yes, and the technique for doing so is called MANOVA (Multiple ANOVA). The assumptions for MANOVA are similar to the ones we have when using Hotelling T2 for two samples: equality of covariance matrices between the groups, the data should come from a multivariate Gaussian distribution for each group, and there should not be outliers.

In this example, we will generate data from a multivariate Gaussian distribution (with three variables) with an arbitrary covariance matrix. We will assign the columns' names to represent the History, Math, and Biology scores...