Why many t tests mislead, how ANOVA works, post hoc comparisons and how the adjustment method changes the answer, limma design matrices and contrasts, batches and repeated measures, and how to choose. With a checklist that remembers what you ticked.
k groups have k(k-1)/2 pairs. Each pair is a chance to be fooled.
| Groups | Pairs | At least one false alarm, all pairwise t tests | One ANOVA |
|---|---|---|---|
| 3 | 3 | 11.3 % | 4.4 % |
| 4 | 6 | 19.9 % | 4.8 % |
| 5 | 10 | 30.5 % | 5.4 % |
| 6 | 15 | 38.2 % | 5.2 % |
Simulation: no group truly differs, 8 samples per group, 3000 runs per row. The 5 percent promise holds for ANOVA, up to simulation error.
One test of the question: is there any difference among the groups?
MSW = SSW / (N - k)Spread of samples around their own group mean.
MSB = SSB / (k - 1)Spread of the group means around the overall mean, weighted by group size.
F = MSB / MSWNear 1 if the groups are the same. Larger when means differ. Compared with an F distribution with k - 1 and N - k degrees of freedom.
Three groups of 8. MSB 14.3, MSW 1.5. At least one group differs, but ANOVA does not say which.
| If | Use |
|---|---|
| Roughly normal, similar spread | One-way ANOVA |
| Unequal variances between groups | Welch ANOVA |
| Skewed data or heavy outliers | Kruskal-Wallis (ranks) |
After a significant ANOVA, find which pairs differ, and adjust for how many you look at.
| Method | Idea | Use when |
|---|---|---|
| Bonferroni | Multiply each p value by the number of comparisons | Few comparisons, simple and conservative. |
| Holm | Stepwise Bonferroni, never less powerful | Flexible, any set of tests. |
| Tukey HSD | Studentized range, built for all pairs | You want every pairwise comparison. |
| Dunnett | Each treatment against one control | Fewer comparisons means more power. |
| Dunn | Rank-based pairwise, with an adjustment | After Kruskal-Wallis. |
| Method | Control vs Treat 1 | Control vs Treat 2 | Treat 1 vs Treat 2 |
|---|---|---|---|
| Unadjusted | 0.039 | 0.0003 | 0.040 |
| Bonferroni | 0.117 | 0.0008 | 0.119 |
| Holm | 0.078 | 0.0008 | 0.078 |
| Tukey HSD | 0.094 | 0.0007 | 0.096 |
| Dunnett | 0.070 | 0.0004 | not tested |
The clear difference survives every method. The borderline pairs flip depending on the method. Choose the method before you look at the data.
Fit one model per feature, then ask questions as contrasts.
A 1 marks the group a sample belongs to. The fit gives a mean per group, per feature.
(-1, +1, 0) asks: is group 2 different from group 1? (0, -1, +1) compares group 3 with group 2.
For every feature. Adjust the p values with Benjamini-Hochberg.
The moderated F test is the ANOVA question for every feature at once.
| Question | Features |
|---|---|
| Treat 1 vs Control | 35 |
| Treat 2 vs Control | 54 |
| Treat 2 vs Treat 1 | 30 |
| F test, any difference | 72 |
Adjusting each contrast separately gave 35 / 54 / 30. Adjusting all contrast p values together gave 36 / 51 / 33. Decide which you will use, and say so.
Add measured nuisance factors to the design so they stop adding noise.
The model estimates and removes the batch shift before testing the groups.
18 samples, 3 groups, a batch effect. Ignoring the batch found about 13 real changes. Including it found about 42, with the false share near 5 percent.
When the same subject appears in several groups, the subject is a term in the design, or a correlation is modelled between measurements on the same subject.
Never adjust for something the treatment itself causes. Balance batches across groups when you plan the study.
| Question | Method |
|---|---|
| One feature, several groups, roughly normal | ANOVA, then Tukey (all pairs) or Dunnett (vs control) |
| One feature, unequal variances | Welch ANOVA, then a pairwise method that allows unequal variances |
| One feature, skewed or outliers | Kruskal-Wallis, then Dunn with an adjustment |
| Thousands of features, log-scale data | limma with contrasts and the F test, then BH |
| Thousands of features, counts | DESeq2 or edgeR contrasts or likelihood ratio test, then BH |
Always adjust across comparisons and across features. State your contrasts before you look at the data, and report the fold change for each one.
Tick as you go. Your ticks stay in this browser.
This series is built from the questions researchers actually ask. Tell us where you got stuck and what you want next. It takes under a minute.
Turn on JavaScript to send your questions, or write to wael@omvitalis.com.
DOIs were checked against Crossref. Check the details against the publisher page before you cite.