Type I and Type II Errors
What are Type I and Type II errors?
Statistical hypothesis testing is about making decisions using data. Data contain inherent variability, therefore, decisions are made with uncertainty, which could lead to errors. A Type I error is finding evidence in favor of the alternative hypothesis when, in fact, the null hypothesis is true. A Type II error is finding evidence against the alternative hypothesis when, in fact, the null hypothesis is false.
A Type I error is also called a false alarm, since it is detecting an effect in the sample only by chance, not because there is a real effect in the population. Other terms for a Type I error are false positive, false discovery, or producer’s risk. In the popular story of the boy who cried wolf, a Type I error is crying “Wolf!” when no wolf is present.
A Type II error is also called a missed opportunity, since it is failing to detect an effect in the sample, even though there is a real effect in the population. Other terms for a Type II error are false negative, miss, or consumer’s risk. In the story of the boy who cried wolf, a Type II error is not seeing the wolf, even though it is there.
How are alpha, beta, and power related?
Even though you can make errors when performing hypothesis tests, you are able to specify the probability of making each type of error. Alpha ($\alpha$) is the probability of making a Type I error, of rejecting the null hypothesis when it’s true. Beta ($\beta$) is the probability of making a Type II error, of failing to reject the null hypothesis when it’s false. When you specify $\alpha$, you specify the significance level of the test. When you specify the power of the test, you specify 1 – $\beta$.
In the figure below, the distribution of the test statistic under the null hypothesis is on the left. The distribution of the test statistic under the alternative hypothesis is on the right. The area under the curve in the right tail of the null distribution is $\alpha$. The area under the curve in the left tail of the alternative distribution, at that same point, is $\beta$. The area under the curve in the right tail of the alternative distribution is power or 1 – $\beta$.
Which error is more important: Type I or Type II?
Neither error is always more important. The more serious error depends on the consequences of the decision. A Type I error is more concerning when acting on a false alarm would be costly, risky, or disruptive. A Type II error is more concerning when missing a real effect would lead to lost opportunity, poor quality, safety concerns, or an ineffective decision.
For example, in a quality inspection setting, rejecting a good product can create unnecessary waste or delay, while accepting a defective product can affect customers or downstream processes. In a medical screening setting, a false positive may lead to unnecessary follow-up, while a false negative may allow a real issue to go unnoticed.
If you decrease the probability of a Type I error, you will increase the probability of a Type II error, assuming the same study design. That is, if you make it harder to detect an effect, you reduce the false alarms, but you will increase missed effects. In practice, decide on a significance level to use in the study, then determine the sample size to obtain required power for the minimum effect size you are interested in, accounting for the variability in the system. This is known as power analysis.
As an example, consider taking a test for strep throat. If the result is positive, that is, the test determines that you have a strep infection, you will likely be prescribed an antibiotic drug. If you really do have the infection, the test worked and you should feel better soon. If the test was a false positive, and you don’t have the infection, you will be taking medicine that you don’t need with its associated costs and side effects. On the other hand, if the test determines that you don’t have a strep infection, you will not be prescribed medicine targeted to that infection. If you really do have the infection, the test was a false negative and your infection could get worse without treatment. If you really don’t have the infection, the correct conclusion was made, and your doctor might order other testing.
For each example below, think about the real-world consequences of each error and decide which one would be more serious in that situation.
Examples of Type I and Type II errors
| Analogy | Type I error | Type II error |
| Smoke alarm | Alarm goes off when there is no fire. | Alarm fails to go off when there is a fire. |
| Medical screening test | Test says a healthy person has the condition. | Test says a sick person does not have the condition. |
| Airport security scanner | Scanner flags a harmless item as dangerous. | Scanner misses a dangerous item. |
| Spam filter | A legitimate email is marked as spam. | A spam email gets through. |
| Quality inspection | A good product is rejected. | A defective product is accepted. |
| Doorbell security camera | An alert notifies you when no one is there. | A lack of an alert fails to notify you when someone is there. |
| Courtroom verdict | An innocent person is found guilty. | A guilty person is found not guilty. |
In practice, a Type I error might create a chance to gather more evidence, while a Type II error might cause you to stop looking. For example, if a smoke alarm sounds when there is no fire, you are likely to investigate and look for confirming evidence. But if the alarm stays silent when there really is a fire, you won’t know that there is anything to investigate. A similar issue can arise in sequential experimentation: an effect that appears significant at one stage may be studied again later, but an effect that appears nonsignificant will likely be dropped from consideration before its true importance is recognized. In sequential studies, you typically don’t confirm negative findings (p > $\alpha$, no difference, no effect), but we do typically retest positive findings (p < $\alpha$).