OmVitalis Academy ▶  Watch Episode 9All guidesCourseEnroll free
Differential Analysis: Thousands of Features
Omics Stats Decoded · Episode 9 · OmVitalis Systems

Differential Analysis: Thousands of Features

What happens when you run one test per feature. Multiple testing and the false discovery rate, Bonferroni, Holm and Benjamini-Hochberg, the volcano plot, limma's moderated t statistic, DESeq2 and edgeR for counts, and how to pick the tool. With a checklist that remembers what you ticked.

The multiple testing problem

A p value below 0.05 accepts a 1 in 20 chance of a false alarm. With many features, those chances add up.

Formula

P(at least one false hit) = 1 - (1 - α)m

m is the number of tests, α the threshold per test.

Example

m = 1 → 5.0%, m = 10 → 40.1%, m = 100 → 99.4%

With 1000 tests, a false hit is certain.

In the video

44 of 1000 null features lit up

Every feature had no real difference, yet 44 had p below 0.05. About 5 percent, as expected.

Takeaway

More features, more luck

Always adjust when you test many features at once.

Corrections

Two different promises: avoid any false alarm, or keep the share of false discoveries small.

MethodRuleControlsUse when
BonferroniReject if p ≤ α / mFamily-wise error rateFew tests, and a single false positive is costly.
HolmSort p. Reject while p(i) ≤ α / (m - i + 1), stop at the first failureFamily-wise error rateAlways at least as powerful as Bonferroni, same guarantee.
Benjamini-HochbergSort p. Find the largest i with p(i) ≤ (i / m) × q. Reject all up to iFalse discovery rateScreening many features. The omics default.
FDR

FDR = false discoveries / all discoveries

Set q to 0.05 and, on average over datasets, about 1 in 20 discoveries is false.

Adjusted p value

The smallest q at which a feature is called

Report adjusted p values or q values next to the raw ones.

Our simulation: 1000 features, 100 truly changed, 6 samples per group

MethodHitsRealFalse
No correction (p < 0.05)1278047
Bonferroni10100
Holm10100
Benjamini-Hochberg, 5%51474

The 5 percent promise holds on average over many datasets. A single run can land a little above or below.

The volcano plot

Size on one axis, certainty on the other.

Horizontal

log2 fold change

How big the change is. Zero means no change.

Vertical

-log10 p value

How sure. Higher means a smaller p value.

Top corners

Large and significant

Use both lines: a corrected significance threshold and a fold change threshold such as 2.

Caution

Heights wobble with few samples

Features with the same fold change sit at different heights because each variance is estimated from few values.

limma and the moderated t

Borrow information across features to stabilise each variance.

Moderated variance

s̃² = (d₀ s₀² + d s²) / (d₀ + d)

s² is this feature's variance with d degrees of freedom. s₀² and d₀ describe the typical variance across all features.

Moderated t

t̃ = difference / ( s̃ · sqrt(1/n₁ + 1/n₂) )

Compared with a t distribution on d₀ + d degrees of freedom.

In the video

200 simulations, 5% FDR

Ordinary t found about 48 of 100 real changes. limma found about 55. False share of discoveries stayed near 5 percent for both.

When it matters

Small samples

The gain is largest with few samples per group. With many samples, moderation changes little.

Our simulation implements the moderated t from the published method. It illustrates the idea and is not a replacement for running limma itself.

Counts: DESeq2 and edgeR

Sequencing counts need depth normalisation and a model for count variance.

IssueWhat the tools do
DepthOne sample sequenced twice as deeply has doubled counts. DESeq2 estimates size factors by the median of ratios. edgeR uses TMM scaling factors.
VariancePoisson variance equals the mean. Real counts are over-dispersed, so both tools use a negative binomial model: variance = mean + dispersion × mean².
Shared informationDispersions are estimated per gene and moderated using all genes, the same idea as limma.
Low countsFold changes of low count genes are noisy. DESeq2 can shrink them toward zero, more strongly when evidence is weak.

The panels in the video are illustrations from a simulation. The tools fit these models gene by gene and test with a Wald or likelihood ratio test (DESeq2) or a quasi-likelihood F test (edgeR).

Choosing the tool

Your dataToolNotes
Continuous values on a log scale: proteomics, metabolomics, arrayslimmaLinear model with moderated t. Supports covariates and paired designs.
RNA-seq countsDESeq2 or edgeRNegative binomial models. Both work well and mostly agree.
RNA-seq counts, limma workflowlimma with voomConverts counts to log values with precision weights.

All of them: fit a model, borrow strength across features, test, and adjust for many tests. With a handful of samples, avoid a plain t test on every feature.

Checklist

Tick as you go. Your ticks stay in this browser.

0 of 0

Before

During

After

Shape the next lesson

This series is built from the questions researchers actually ask. Tell us where you got stuck and what you want next. It takes under a minute.

Turn on JavaScript to send your questions, or write to wael@omvitalis.com.

References

DOIs were checked against Crossref. Check the details against the publisher page before you cite.

  1. False discovery rateBenjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc B 1995;57:289-300.doi:10.1111/j.2517-6161.1995.tb02031.x
  2. Holm's step-down methodHolm S. A simple sequentially rejective multiple test procedure. Scand J Stat 1979;6:65-70. No DOI.
  3. q valuesStorey JD, Tibshirani R. Statistical significance for genomewide studies. Proc Natl Acad Sci USA 2003;100:9440-9445.doi:10.1073/pnas.1530509100
  4. The moderated tSmyth GK. Linear models and empirical Bayes methods for assessing differential expression in microarray experiments. Stat Appl Genet Mol Biol 2004;3:Article 3.doi:10.2202/1544-6115.1027
  5. The limma packageRitchie ME, et al. limma powers differential expression analyses for RNA-sequencing and microarray studies. Nucleic Acids Res 2015;43:e47.doi:10.1093/nar/gkv007
  6. voomLaw CW, Chen Y, Shi W, Smyth GK. voom: precision weights unlock linear model analysis tools for RNA-seq read counts. Genome Biol 2014;15:R29.doi:10.1186/gb-2014-15-2-r29
  7. DESeq2Love MI, Huber W, Anders S. Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2. Genome Biol 2014;15:550.doi:10.1186/s13059-014-0550-8
  8. Size factors, median of ratiosAnders S, Huber W. Differential expression analysis for sequence count data. Genome Biol 2010;11:R106.doi:10.1186/gb-2010-11-10-r106
  9. edgeRRobinson MD, McCarthy DJ, Smyth GK. edgeR: a Bioconductor package for differential expression analysis of digital gene expression data. Bioinformatics 2010;26:139-140.doi:10.1093/bioinformatics/btp616
  10. Fold change shrinkageZhu A, Ibrahim JG, Love MI. Heavy-tailed prior distributions for sequence count data: removing the noise and preserving large differences. Bioinformatics 2019;35:2084-2092.doi:10.1093/bioinformatics/bty895