What problem arises when performing multiple post-hoc pairwise comparisons after ANOVA?

What problem arises when performing multiple post-hoc pairwise comparisons after ANOVA?

• A. Loss of power
• B. Alpha inflation
• C. Decreased sample size
• D. Bland-Altman effect

Answer: (B)

When you do additional pairwise comparisons after a significant ANOVA, the chance of finding at least one false positive across all those tests grows. This is alpha inflation—the familywise error rate goes up as more tests are performed. Each test carries its own small risk of a Type I error, and the cumulative risk across the full set of comparisons increases with the number of comparisons. A helpful way to see it is to imagine four groups: there are six possible pairwise comparisons. If each test uses a 5% threshold, the probability of getting at least one false positive among those six comparisons is 1 − 0.95^6, which is about 26%. To keep the overall false-positive rate under control, researchers apply adjustments (like Bonferroni or Holm corrections) or use methods designed for all pairwise comparisons (like Tukey’s test) that control the familywise error rate. The other options don’t fit as the main issue here: decreased sample size isn’t affected by doing post-hoc tests, Bland-Altman is about agreement between two measurement methods, and loss of power is a separate concern that arises when overly strict corrections are applied, not the core problem posed by performing multiple tests.

When you do additional pairwise comparisons after a significant ANOVA, the chance of finding at least one false positive across all those tests grows. This is alpha inflation—the familywise error rate goes up as more tests are performed. Each test carries its own small risk of a Type I error, and the cumulative risk across the full set of comparisons increases with the number of comparisons.

A helpful way to see it is to imagine four groups: there are six possible pairwise comparisons. If each test uses a 5% threshold, the probability of getting at least one false positive among those six comparisons is 1 − 0.95^6, which is about 26%. To keep the overall false-positive rate under control, researchers apply adjustments (like Bonferroni or Holm corrections) or use methods designed for all pairwise comparisons (like Tukey’s test) that control the familywise error rate.

The other options don’t fit as the main issue here: decreased sample size isn’t affected by doing post-hoc tests, Bland-Altman is about agreement between two measurement methods, and loss of power is a separate concern that arises when overly strict corrections are applied, not the core problem posed by performing multiple tests.