A/B Test Significance Calculator
That "10% lift" in your experiment — is it real or random noise?
Why "10% More Conversions" Is Usually Noise
The most common A/B testing mistake: declaring a winner too early. With small sample sizes, a 10% lift can easily be random variation. You need statistical significance (typically p < 0.05, or 95% confidence) before trusting a result.
For a baseline 5% conversion rate and a 10% relative lift (to 5.5%), you need approximately 28,000 visitors per group. Most teams stop their tests far too early.
Sample Size Requirements
Minimum visitors per group needed to detect a lift at 95% confidence / 80% power:
| Baseline Rate | 5% Lift | 10% Lift | 20% Lift |
|---|---|---|---|
| 1% | 635K | 159K | 40K |
| 2% | 312K | 78K | 20K |
| 5% | 118K | 30K | 7.5K |
| 10% | 54K | 14K | 3.5K |
| 20% | 24K | 6.1K | 1.6K |
This is why low-traffic sites struggle with A/B testing — detecting small lifts requires enormous samples.
Common A/B Testing Mistakes
Peeking: Checking results daily and stopping when they look good inflates false positive rates to 25%+ (from the target 5%).
Multiple variants: Testing 5 variants without correction means a 23% chance of a false winner.
Seasonal effects: A test running Monday-Friday misses weekend behavior. Always run full-week multiples.