OmVitalis Academy ▶  Watch Episode 8All guidesCourseEnroll free
Differential Analysis: Two Groups
Omics Stats Decoded · Episode 8 · OmVitalis Systems

Differential Analysis: Two Groups

How to compare one feature between two groups. The t statistic as signal over noise, how fold change is calculated, how to read a box plot, paired versus unpaired designs, what a p value really says, and how to pick the test. With a checklist that remembers what you ticked.

Signal and noise

Every two-group test asks one question: is the difference between the groups large compared with the spread inside them?

Signal

difference = mean B - mean A

How far apart the group averages sit.

Noise

SE = sqrt(s²A/nA + s²B/nB)

The standard error of that difference. More spread or fewer samples make it larger.

Statistic

t = difference / SE

In the video: 1.38 / 0.76 = 1.82 when the groups were noisy, and 3.55 for the real data. Same difference, different noise, different certainty.

Then

t becomes a p value

The t distribution converts the statistic into the chance of seeing a value this large if there is no real difference.

Fold change

How big is the difference, in a form people can read.

QuantityHow to compute itExample
log2 fold changemean of group B minus mean of group A, both on the log2 scale21.26 - 19.89 = 1.38
Fold change2 raised to the log2 fold change21.38 = 2.6
DirectionPositive means higher in B, negative means lower in B+1 is double, -1 is half
Why logs

Symmetry

On the ratio scale, doubling is +1.0 and halving is -0.5. On the log2 scale they are +1 and -1.

Watch out

Subtract on the log scale

Averaging raw values first lets extreme samples pull the ratio. In our example it gave 3.15 instead of 2.6.

Remember

Size is not certainty

The fold change says how big. The p value says how sure. A feature needs both to be interesting.

Counts data

Different tools

For RNA-seq counts, packages such as DESeq2 and edgeR estimate fold change with count models. Episode 9 covers them.

Reading a box plot

A box plot summarises the middle of the data and flags what lies far from it.

PartWhat it shows
Line in the boxThe median: half the samples above, half below.
BoxQ1 to Q3, the middle 50 percent of samples. Its height is the IQR.
WhiskersThe furthest sample within 1.5 × IQR of the box.
Dots beyondPossible outliers by the 1.5 × IQR rule.
Not shownThe mean and the number of samples. Overlay the individual points.

In our example the treated group has one dot far above its box (23.58). The box plot shows it, the t test is pulled by it, and a rank-based test barely notices it.

Paired or unpaired?

Ask whether each sample in one group has a partner in the other.

Unpaired

Different subjects in each group

Use Welch's t test. It compares group averages against the spread within groups.

Paired

The same subject measured twice

Before and after, tumour and adjacent normal, left and right. Test the one-line differences, so the subject to subject variation cancels.

In the video

Same data, two answers

Unpaired: t = 0.51, p = 0.619. Paired: t = 2.81, p = 0.020. Eight of the ten lines went up.

Rule

Decide from the design

Pairing is a property of how the study was run, not a choice made after you see the p value.

What a p value means

The chance of a difference at least this large, if there were no real difference between the groups.

Permutation view

Shuffle the labels

If the groups do not differ, the labels are arbitrary. Shuffle them thousands of times and record the difference each time.

Reading it

Where does your data land?

The p value is the share of shuffles at least as extreme as the real difference. In the video, about 0.001 from 5000 shuffles.

Not

Not the chance the treatment does nothing

It assumes no effect and asks how surprising the data are. It does not give the probability that the hypothesis is true.

Reporting

Report p with the effect size

Give the fold change, an interval and the p value. A permutation p value is never exactly zero: report it as at least 1 / (number of shuffles + 1).

Choosing the test

Choose by the design and the shape of the data, before you look at the p value.

SituationTestWhy
Two independent groups, roughly bell shapedWelch t testDoes not assume equal spread. A safe default.
The same subject in both conditionsPaired t testRemoves subject to subject variation.
Skewed data, heavy outliers, ordinal scoresMann-Whitney (unpaired) or Wilcoxon signed rank (paired)Compares ranks, so extreme values have little pull.

What our simulation showed

QuestionResult
False positives at 5 percent, unequal spread and unequal group sizes: Student's t21.9 percent
Same setting: Welch's t5.2 percent
Power on skewed data: t test26.9 percent
Power on skewed data: Mann-Whitney45.0 percent

Student's test assumes both groups have the same spread. When they do not, its error rate drifts away from 5 percent. Welch's version stays close to it.

Checklist

Tick as you go. Your ticks stay in this browser.

0 of 0

Before

During

After

Shape the next lesson

This series is built from the questions researchers actually ask. Tell us where you got stuck and what you want next. It takes under a minute.

Turn on JavaScript to send your questions, or write to wael@omvitalis.com.

References

DOIs were checked against Crossref. Check the details against the publisher page before you cite.

  1. The unequal variance t testWelch BL. The generalization of "Student's" problem when several different population variances are involved. Biometrika 1947;34:28-35.doi:10.1093/biomet/34.1-2.28
  2. Why Welch by defaultRuxton GD. The unequal variance t-test is an underused alternative to Student's t-test and the Mann-Whitney U test. Behav Ecol 2006;17:688-690.doi:10.1093/beheco/ark016
  3. Permutation p valuesPhipson B, Smyth GK. Permutation p-values should never be zero: calculating exact p-values when permutations are randomly drawn. Stat Appl Genet Mol Biol 2010;9:Article 39.doi:10.2202/1544-6115.1585
  4. What p values meanWasserstein RL, Lazar NA. The ASA statement on p-values: context, process, and purpose. Am Stat 2016;70:129-133.doi:10.1080/00031305.2016.1154108
  5. Box plotsKrzywinski M, Altman N. Visualizing samples with box plots. Nat Methods 2014;11:119-120.doi:10.1038/nmeth.2813
  6. The 1.5 × IQR ruleTukey JW. Exploratory Data Analysis. Addison-Wesley; 1977. Book, no DOI.