Reference Distributions

What is a reference distribution?

In hypothesis testing, a reference distribution is the probability distribution of the test statistic that assumes that the null hypothesis is true. You can use the reference distribution to calculate the p-value of the test. The p-value is the probability under that reference distribution of obtaining a test statistic at least as large as the one you did if the null hypothesis is true.

Why are reference distributions important?

Without a reference distribution, the calculated test statistic is just a number that is impossible to judge. The reference distribution supplies the context to judge whether the test statistic is in the noise or has enough signal to supply evidence for the alternative hypothesis.

The figure below shows the results of three hypothesis tests, all with the same calculated test statistic. The value of the test statistic is 3.4 for all three tests and is represented with a red line. The critical value of the reference distribution depends on the degrees of freedom and significance level (here, $\alpha$ = 0.05) and is represented by a blue line. Different hypothesis tests with different sample sizes have different critical values. Whether that hypothesis test is deemed significant or not depends on the reference distribution.

The first graph represents a one-samplet-test with 12 df. It is significant because the red line at 3.4 is much larger than the blue line at the critical value of 1.78. The middle graph represents a chi-square test of independence for a contingency table with 3 df. It is not significant because the red line at 3.4 is much smaller than the critical value of 7.81. The final graph represents an F-test in an ANOVA table with (10, 8) df. It is weakly significant because the red line at 3.4 is very close to the critical value of 3.34.

Three reference distributions, all with the same test statistic, but different critical values.

What are some common reference distributions?

Many reference distributions are indexed by their degrees of freedom, which are related to the sample size of the data being collected for the hypothesis test.

Common reference distributions include the following:

t distribution: Used when testing hypotheses about one mean, two means or a slope. The t distribution is symmetric like the normal distribution, but has heavier tails, allowing for extra uncertainty from estimating variability.
t reference distributions indexed by degrees of freedom.
Chi-square ($\chi^2$) distribution: Used for tests involving sums of squared quantities. It commonly appears in tests of independence, goodness-of-fit tests, and inference about variances. Because it is based on squared values, the chi-square distribution is nonnegative and right-skewed.
reference-distribution-pic3.png
F distribution: Used for tests based on ratios of variances, like in an ANOVA table. It is central to analysis of variance, regression model comparisons, and tests for equality of variances. The F distribution is also nonnegative and right-skewed.
reference-distribution-pic4.png