Reference Distributions
What is a reference distribution?
In hypothesis testing, a reference distribution is the probability distribution of the test statistic that assumes that the null hypothesis is true. You can use the reference distribution to calculate the p-value of the test. The p-value is the probability under that reference distribution of obtaining a test statistic at least as large as the one you did if the null hypothesis is true.
Why are reference distributions important?
Without a reference distribution, the calculated test statistic is just a number that is impossible to judge. The reference distribution supplies the context to judge whether the test statistic is in the noise or has enough signal to supply evidence for the alternative hypothesis.
The figure below shows the results of three hypothesis tests, all with the same calculated test statistic. The value of the test statistic is 3.4 for all three tests and is represented with a red line. The critical value of the reference distribution depends on the degrees of freedom and significance level (here, $\alpha$ = 0.05) and is represented by a blue line. Different hypothesis tests with different sample sizes have different critical values. Whether that hypothesis test is deemed significant or not depends on the reference distribution.
The first graph represents a one-samplet-test with 12 df. It is significant because the red line at 3.4 is much larger than the blue line at the critical value of 1.78. The middle graph represents a chi-square test of independence for a contingency table with 3 df. It is not significant because the red line at 3.4 is much smaller than the critical value of 7.81. The final graph represents an F-test in an ANOVA table with (10, 8) df. It is weakly significant because the red line at 3.4 is very close to the critical value of 3.34.
What are some common reference distributions?
Many reference distributions are indexed by their degrees of freedom, which are related to the sample size of the data being collected for the hypothesis test.
Common reference distributions include the following: