What happens when you run one test per feature. Multiple testing and the false discovery rate, Bonferroni, Holm and Benjamini-Hochberg, the volcano plot, limma's moderated t statistic, DESeq2 and edgeR for counts, and how to pick the tool. With a checklist that remembers what you ticked.
A p value below 0.05 accepts a 1 in 20 chance of a false alarm. With many features, those chances add up.
P(at least one false hit) = 1 - (1 - α)mm is the number of tests, α the threshold per test.
With 1000 tests, a false hit is certain.
Every feature had no real difference, yet 44 had p below 0.05. About 5 percent, as expected.
Always adjust when you test many features at once.
Two different promises: avoid any false alarm, or keep the share of false discoveries small.
| Method | Rule | Controls | Use when |
|---|---|---|---|
| Bonferroni | Reject if p ≤ α / m | Family-wise error rate | Few tests, and a single false positive is costly. |
| Holm | Sort p. Reject while p(i) ≤ α / (m - i + 1), stop at the first failure | Family-wise error rate | Always at least as powerful as Bonferroni, same guarantee. |
| Benjamini-Hochberg | Sort p. Find the largest i with p(i) ≤ (i / m) × q. Reject all up to i | False discovery rate | Screening many features. The omics default. |
FDR = false discoveries / all discoveriesSet q to 0.05 and, on average over datasets, about 1 in 20 discoveries is false.
Report adjusted p values or q values next to the raw ones.
| Method | Hits | Real | False |
|---|---|---|---|
| No correction (p < 0.05) | 127 | 80 | 47 |
| Bonferroni | 10 | 10 | 0 |
| Holm | 10 | 10 | 0 |
| Benjamini-Hochberg, 5% | 51 | 47 | 4 |
The 5 percent promise holds on average over many datasets. A single run can land a little above or below.
Size on one axis, certainty on the other.
How big the change is. Zero means no change.
How sure. Higher means a smaller p value.
Use both lines: a corrected significance threshold and a fold change threshold such as 2.
Features with the same fold change sit at different heights because each variance is estimated from few values.
Borrow information across features to stabilise each variance.
s̃² = (d₀ s₀² + d s²) / (d₀ + d)s² is this feature's variance with d degrees of freedom. s₀² and d₀ describe the typical variance across all features.
t̃ = difference / ( s̃ · sqrt(1/n₁ + 1/n₂) )Compared with a t distribution on d₀ + d degrees of freedom.
Ordinary t found about 48 of 100 real changes. limma found about 55. False share of discoveries stayed near 5 percent for both.
The gain is largest with few samples per group. With many samples, moderation changes little.
Our simulation implements the moderated t from the published method. It illustrates the idea and is not a replacement for running limma itself.
Sequencing counts need depth normalisation and a model for count variance.
| Issue | What the tools do |
|---|---|
| Depth | One sample sequenced twice as deeply has doubled counts. DESeq2 estimates size factors by the median of ratios. edgeR uses TMM scaling factors. |
| Variance | Poisson variance equals the mean. Real counts are over-dispersed, so both tools use a negative binomial model: variance = mean + dispersion × mean². |
| Shared information | Dispersions are estimated per gene and moderated using all genes, the same idea as limma. |
| Low counts | Fold changes of low count genes are noisy. DESeq2 can shrink them toward zero, more strongly when evidence is weak. |
The panels in the video are illustrations from a simulation. The tools fit these models gene by gene and test with a Wald or likelihood ratio test (DESeq2) or a quasi-likelihood F test (edgeR).
| Your data | Tool | Notes |
|---|---|---|
| Continuous values on a log scale: proteomics, metabolomics, arrays | limma | Linear model with moderated t. Supports covariates and paired designs. |
| RNA-seq counts | DESeq2 or edgeR | Negative binomial models. Both work well and mostly agree. |
| RNA-seq counts, limma workflow | limma with voom | Converts counts to log values with precision weights. |
All of them: fit a model, borrow strength across features, test, and adjust for many tests. With a handful of samples, avoid a plain t test on every feature.
Tick as you go. Your ticks stay in this browser.
This series is built from the questions researchers actually ask. Tell us where you got stuck and what you want next. It takes under a minute.
Turn on JavaScript to send your questions, or write to wael@omvitalis.com.
DOIs were checked against Crossref. Check the details against the publisher page before you cite.