Home›Glossary›Statistical Significance

Statistical Significance

General

Statistical Significance (p-value / Confidence Level)

A measure of how likely it is that an observed difference between two groups (such as A/B test variants) is real rather than due to random chance — typically expressed as a p-value or confidence level.

Definition

Statistical significance is a measure of confidence that an observed difference between two groups — such as the conversion rates of variant A and variant B in an A/B test — reflects a real effect rather than random noise in the sample. It is most commonly expressed as a p-value (the probability of seeing this result, or a more extreme one, if there were truly no difference) or as a confidence level (1 minus the p-value, expressed as a percentage).

Statistical significance is the foundation of rigorous A/B testing and experimentation: without it, teams risk declaring a "winning" variant that was actually just a random fluctuation, leading to decisions based on noise rather than real user behaviour.

Formula

Pooled Conversion Rate = (Conversions A + Conversions B) / (Visitors A + Visitors B)

Standard Error = √[Pooled Rate × (1 − Pooled Rate) × (1/Visitors A + 1/Visitors B)]

Z-Score = (Conversion Rate A − Conversion Rate B) / Standard Error

p-value is derived from the Z-score using the standard normal distribution — a common threshold for declaring significance is p ≤ 0.05 (95% confidence level).

Worked Example

An e-commerce site runs an A/B test on its checkout page:

Variant Visitors Conversions Conversion Rate
A (control) 5,000 250 5.0%
B (new design) 5,000 290 5.8%

Relative Uplift = (5.8% − 5.0%) / 5.0% = 16%

Running the Z-test on these numbers might produce a Z-score around 1.9, corresponding to a p-value of roughly 0.057 — just above the conventional 0.05 threshold, meaning this result would not be declared statistically significant despite the visible 16% lift. The test would need either a larger sample or a longer run to confirm whether the effect is real.

Use the A/B Test Significance calculator to check your own test results before declaring a winner.

Key Things to Know

  • Significance is not the same as importance: A statistically significant 0.1% lift in conversion rate on a massive sample might not be worth the engineering effort to ship, while a large but not-yet-significant lift might be worth extending the test to confirm.
  • Pre-commit to a sample size before launching the test: Calculating the required sample size in advance — based on baseline conversion rate and minimum detectable effect — prevents both wasted traffic (testing too long after the answer is clear) and false positives (stopping too early on a lucky streak).
  • Segment results carefully, and correct for multiple comparisons: Running the same significance test across many segments (device type, traffic source, country) multiplies the chance of a false positive somewhere in the data purely by chance — a single significant result among twenty segment cuts is exactly what you'd expect from noise alone, not a finding to act on without further validation.
  • Higher confidence thresholds require larger samples: Moving from a 95% to a 99% confidence requirement substantially increases the sample size needed to detect the same effect size, trading a lower false-positive rate for a longer (or larger) test.
  • Novelty effects can fade after the test ends: A new design sometimes wins during a test partly because it's new and attention-grabbing, then the lift shrinks once the novelty wears off — long-running tests or post-launch monitoring help confirm a win holds up over time.

Related Calculators

A/B Test Significance Calculator →Check if your A/B test results are statistically significant. Enter visitors and conversions for both variants to get conversion rates, uplift, p-value, and verdict.
Conversion Rate Calculator →Calculate your website or campaign conversion rate instantly. Enter total visitors and conversions to find your rate and benchmark it against industry averages.
Z-Score Calculator →Calculate z-score, percentile rank, and probability for any data point. Enter value, mean, and standard deviation to find where it falls in the distribution. Free online tool.
Sales Funnel Calculator →Calculate conversion rates at every stage of your sales funnel — leads to MQL, MQL to SQL, SQL to opportunity, and opportunity to close. Identify your biggest funnel drop-off instantly.

Related Terms

Conversion RateConversion RateStandard DeviationStatistical Measure of Data VariabilityROASReturn on Ad Spend

Frequently Asked Questions

A p-value of 0.05 means there's a 5% probability of observing a difference as large as the one measured (or larger) purely by random chance, if there were truly no difference between the two groups. It does not mean there's a 95% probability your variant is better, and it doesn't measure the size or business importance of the effect — only how unlikely the observed pattern is to be a fluke.
Required sample size depends on three things: the baseline conversion rate, the minimum effect size you want to detect, and your desired confidence level (commonly 95%). Smaller baseline conversion rates and smaller effect sizes both require larger samples. As a rough guide, detecting a relative lift of 10-20% on a baseline conversion rate of a few percent often requires several thousand visitors per variant — use a sample size calculator before launching a test, not after.
No — stopping a test as soon as it crosses the significance threshold (a practice known as 'peeking') dramatically inflates the false positive rate, because random fluctuations will cross the 95% threshold by chance at some point during almost any test if you check repeatedly. Decide on a sample size or test duration in advance and only evaluate significance once that predetermined point is reached, or use a sequential testing method designed for early stopping.
A result can be statistically significant (very unlikely to be chance) but practically insignificant (the actual effect size is too small to matter for the business) — common with very large sample sizes, which can detect tiny, meaningless differences as 'significant.' Conversely, a genuinely important effect can fail to reach statistical significance simply because the sample size was too small. Always look at both the p-value and the magnitude of the lift.
A Type I error (false positive) is concluding there's a real difference when there isn't one — controlled by your significance threshold (a 5% threshold means a 5% Type I error rate). A Type II error (false negative) is failing to detect a real difference that does exist — usually caused by an underpowered test (too small a sample size). Reducing one type of error without increasing sample size typically increases the other, which is why both significance level and statistical power must be planned together before running a test.