OmVitalis Academy ▶  Watch Episode 10All guidesCourseEnroll free
Differential Analysis: Three or More Groups
Omics Stats Decoded · Episode 10 · OmVitalis Systems

Differential Analysis: Three or More Groups

Why many t tests mislead, how ANOVA works, post hoc comparisons and how the adjustment method changes the answer, limma design matrices and contrasts, batches and repeated measures, and how to choose. With a checklist that remembers what you ticked.

Why not many t tests?

k groups have k(k-1)/2 pairs. Each pair is a chance to be fooled.

GroupsPairsAt least one false alarm, all pairwise t testsOne ANOVA
3311.3 %4.4 %
4619.9 %4.8 %
51030.5 %5.4 %
61538.2 %5.2 %

Simulation: no group truly differs, 8 samples per group, 3000 runs per row. The 5 percent promise holds for ANOVA, up to simulation error.

ANOVA

One test of the question: is there any difference among the groups?

Within

MSW = SSW / (N - k)

Spread of samples around their own group mean.

Between

MSB = SSB / (k - 1)

Spread of the group means around the overall mean, weighted by group size.

Statistic

F = MSB / MSW

Near 1 if the groups are the same. Larger when means differ. Compared with an F distribution with k - 1 and N - k degrees of freedom.

In the video

F = 9.7, p = 0.001

Three groups of 8. MSB 14.3, MSW 1.5. At least one group differs, but ANOVA does not say which.

IfUse
Roughly normal, similar spreadOne-way ANOVA
Unequal variances between groupsWelch ANOVA
Skewed data or heavy outliersKruskal-Wallis (ranks)

Post hoc comparisons

After a significant ANOVA, find which pairs differ, and adjust for how many you look at.

MethodIdeaUse when
BonferroniMultiply each p value by the number of comparisonsFew comparisons, simple and conservative.
HolmStepwise Bonferroni, never less powerfulFlexible, any set of tests.
Tukey HSDStudentized range, built for all pairsYou want every pairwise comparison.
DunnettEach treatment against one controlFewer comparisons means more power.
DunnRank-based pairwise, with an adjustmentAfter Kruskal-Wallis.

Same data, five answers (adjusted p values)

MethodControl vs Treat 1Control vs Treat 2Treat 1 vs Treat 2
Unadjusted0.0390.00030.040
Bonferroni0.1170.00080.119
Holm0.0780.00080.078
Tukey HSD0.0940.00070.096
Dunnett0.0700.0004not tested

The clear difference survives every method. The borderline pairs flip depending on the method. Choose the method before you look at the data.

limma: design matrix and contrasts

Fit one model per feature, then ask questions as contrasts.

Design matrix

One row per sample, one column per group

A 1 marks the group a sample belongs to. The fit gives a mean per group, per feature.

Contrast

A recipe that combines means

(-1, +1, 0) asks: is group 2 different from group 1? (0, -1, +1) compares group 3 with group 2.

Per contrast

Fold change, moderated t, p value

For every feature. Adjust the p values with Benjamini-Hochberg.

F test

Anything different?

The moderated F test is the ANOVA question for every feature at once.

Our simulation: 1000 features, 4 samples per group, BH at 5%

QuestionFeatures
Treat 1 vs Control35
Treat 2 vs Control54
Treat 2 vs Treat 130
F test, any difference72

Adjusting each contrast separately gave 35 / 54 / 30. Adjusting all contrast p values together gave 36 / 51 / 33. Decide which you will use, and say so.

Batches and repeated measures

Add measured nuisance factors to the design so they stop adding noise.

Batch

Add a batch column

The model estimates and removes the batch shift before testing the groups.

In the video

13 versus 42 of 100

18 samples, 3 groups, a batch effect. Ignoring the batch found about 13 real changes. Including it found about 42, with the false share near 5 percent.

Subjects

Repeated measures

When the same subject appears in several groups, the subject is a term in the design, or a correlation is modelled between measurements on the same subject.

Caution

Only what you measured

Never adjust for something the treatment itself causes. Balance batches across groups when you plan the study.

Choosing the method

QuestionMethod
One feature, several groups, roughly normalANOVA, then Tukey (all pairs) or Dunnett (vs control)
One feature, unequal variancesWelch ANOVA, then a pairwise method that allows unequal variances
One feature, skewed or outliersKruskal-Wallis, then Dunn with an adjustment
Thousands of features, log-scale datalimma with contrasts and the F test, then BH
Thousands of features, countsDESeq2 or edgeR contrasts or likelihood ratio test, then BH

Always adjust across comparisons and across features. State your contrasts before you look at the data, and report the fold change for each one.

Checklist

Tick as you go. Your ticks stay in this browser.

0 of 0

Before

During

After

Shape the next lesson

This series is built from the questions researchers actually ask. Tell us where you got stuck and what you want next. It takes under a minute.

Turn on JavaScript to send your questions, or write to wael@omvitalis.com.

References

DOIs were checked against Crossref. Check the details against the publisher page before you cite.

  1. Tukey HSDTukey JW. Comparing individual means in the analysis of variance. Biometrics 1949;5:99-114.doi:10.2307/3001913
  2. Dunnett's testDunnett CW. A multiple comparison procedure for comparing several treatments with a control. J Am Stat Assoc 1955;50:1096-1121.doi:10.1080/01621459.1955.10501294
  3. Kruskal-WallisKruskal WH, Wallis WA. Use of ranks in one-criterion variance analysis. J Am Stat Assoc 1952;47:583-621.doi:10.1080/01621459.1952.10483441
  4. Dunn's testDunn OJ. Multiple comparisons using rank sums. Technometrics 1964;6:241-252.doi:10.1080/00401706.1964.10490181
  5. Welch ANOVAWelch BL. On the comparison of several mean values: an alternative approach. Biometrika 1951;38:330-336.doi:10.1093/biomet/38.3-4.330
  6. Holm's methodHolm S. A simple sequentially rejective multiple test procedure. Scand J Stat 1979;6:65-70. No DOI.
  7. False discovery rateBenjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc B 1995;57:289-300.doi:10.1111/j.2517-6161.1995.tb02031.x
  8. Moderated testsSmyth GK. Linear models and empirical Bayes methods for assessing differential expression in microarray experiments. Stat Appl Genet Mol Biol 2004;3:Article 3.doi:10.2202/1544-6115.1027
  9. The limma packageRitchie ME, et al. limma powers differential expression analyses for RNA-sequencing and microarray studies. Nucleic Acids Res 2015;43:e47.doi:10.1093/nar/gkv007