Home›Glossary›Statistical Significance

Statistical Significance

General

Statistical Significance (p-value / Confidence Level)

A measure of how likely it is that an observed difference between two groups (such as A/B test variants) is real rather than due to random chance — typically expressed as a p-value or confidence level.

Definition

Statistical significance is a measure of confidence that an observed difference between two groups, such as the conversion rates of variant A and variant B in an A/B test, reflects a real effect rather than random noise in the sample. It is most commonly expressed as a p-value (the probability of seeing this result, or a more extreme one, if there were truly no difference) or as a confidence level (1 minus the p-value, expressed as a percentage).

Statistical significance is the foundation of rigorous A/B testing and experimentation: without it, teams risk declaring a "winning" variant that was actually just a random fluctuation, leading to decisions based on noise rather than real user behaviour.

Formula

Pooled Conversion Rate = (Conversions A + Conversions B) / (Visitors A + Visitors B)

Standard Error = √[Pooled Rate × (1 − Pooled Rate) × (1/Visitors A + 1/Visitors B)]

Z-Score = (Conversion Rate A − Conversion Rate B) / Standard Error

p-value is derived from the Z-score using the standard normal distribution, a common threshold for declaring significance is p ≤ 0.05 (95% confidence level).

Worked Example

An e-commerce site runs an A/B test on its checkout page:

Variant Visitors Conversions Conversion Rate
A (control) 5,000 250 5.0%
B (new design) 5,000 290 5.8%

Relative Uplift = (5.8% − 5.0%) / 5.0% = 16%

Running the Z-test on these numbers might produce a Z-score around 1.9, corresponding to a p-value of roughly 0.057, just above the conventional 0.05 threshold, meaning this result would not be declared statistically significant despite the visible 16% lift. The test would need either a larger sample or a longer run to confirm whether the effect is real.

Use the A/B Test Significance calculator to check your own test results before declaring a winner.

Key Things to Know

  • Significance is not the same as importance: A statistically significant 0.1% lift in conversion rate on a massive sample might not be worth the engineering effort to ship, while a large but not-yet-significant lift might be worth extending the test to confirm.
  • Pre-commit to a sample size before launching the test: Calculating the required sample size in advance, based on baseline conversion rate and minimum detectable effect, prevents both wasted traffic (testing too long after the answer is clear) and false positives (stopping too early on a lucky streak).
  • Segment results carefully, and correct for multiple comparisons: Running the same significance test across many segments (device type, traffic source, country) multiplies the chance of a false positive somewhere in the data purely by chance, a single significant result among twenty segment cuts is exactly what you'd expect from noise alone, not a finding to act on without further validation.
  • Higher confidence thresholds require larger samples: Moving from a 95% to a 99% confidence requirement substantially increases the sample size needed to detect the same effect size, trading a lower false-positive rate for a longer (or larger) test.
  • Novelty effects can fade after the test ends: A new design sometimes wins during a test partly because it's new and attention-grabbing, then the lift shrinks once the novelty wears off, long-running tests or post-launch monitoring help confirm a win holds up over time.

Related Calculators

A/B Test Significance Calculator →Check if your A/B test results are statistically significant. Enter visitors and conversions for both variants to get conversion rates and verdict.
Conversion Rate Calculator →Calculate your website or campaign conversion rate instantly. Enter total visitors and conversions to find your rate and benchmark it against industry averages.
Z-Score Calculator →Calculate z-score, percentile rank, and probability for any data point. Enter value, mean, and standard deviation to find its distribution position.
Sales Funnel Calculator →Calculate conversion rates at every stage of your sales funnel — leads to MQL, MQL to SQL, SQL to opportunity — to find your biggest drop-off.

Related Terms

Conversion RateConversion RateStandard DeviationStatistical Measure of Data VariabilityROASReturn on Ad Spend

Frequently Asked Questions

What does a p-value of 0.05 actually mean?
A p-value of 0.05 means there's a 5% probability of observing a difference as large as the one measured (or larger) purely by random chance, if there were truly no difference between the two groups. It does not mean there's a 95% probability your variant is better, and it doesn't measure the size or business importance of the effect, only how unlikely the observed pattern is to be a fluke.
What sample size do I need for a significant A/B test?
Required sample size depends on three things: the baseline conversion rate, the minimum effect size you want to detect, and your desired confidence level (commonly 95%). Smaller baseline conversion rates and smaller effect sizes both require larger samples. As a rough guide, detecting a relative lift of 10-20% on a baseline conversion rate of a few percent often requires several thousand visitors per variant, use a sample size calculator before launching a test, not after.
Can I stop an A/B test early once it looks significant?
No, stopping a test as soon as it crosses the significance threshold (a practice known as 'peeking') dramatically inflates the false positive rate, because random fluctuations will cross the 95% threshold by chance at some point during almost any test if you check repeatedly. Decide on a sample size or test duration in advance and only evaluate significance once that predetermined point is reached, or use a sequential testing method designed for early stopping.
What is the difference between statistical significance and practical significance?
A result can be statistically significant (very unlikely to be chance) but practically insignificant (the actual effect size is too small to matter for the business), common with very large sample sizes, which can detect tiny, meaningless differences as 'significant.' Conversely, a genuinely important effect can fail to reach statistical significance simply because the sample size was too small. Always look at both the p-value and the magnitude of the lift.
What is a Type I error and a Type II error in A/B testing?
A Type I error (false positive) is concluding there's a real difference when there isn't one, controlled by your significance threshold (a 5% threshold means a 5% Type I error rate). A Type II error (false negative) is failing to detect a real difference that does exist, usually caused by an underpowered test (too small a sample size). Reducing one type of error without increasing sample size typically increases the other, which is why both significance level and statistical power must be planned together before running a test.