Home›Articles›How To›A/B Test Significance
HOW TO

How to Run an A/B Test Significance Test

Calculate A/B test significance step by step using conversion rates, sample sizes, and the z-test formula, plus the most common mistakes that lead to false positives.

Updated 2026-06-29

Free calculators used in this guide

A/B Test Significance CalculatorConversion Rate CalculatorZ-Score CalculatorSales Funnel Calculator

Overview

Before declaring a winner in an A/B test, you need to know whether the observed difference between variants is a real effect or just random noise in the sample. This article walks through exactly how to calculate statistical significance for a two-variant test — the z-test formula, a worked example, and the most common mistakes that lead teams to false positives.

This guide is for product managers, growth marketers, and anyone running conversion rate experiments who needs to interpret test results correctly before shipping a change.

What You Need

Before calculating significance, gather:

  • Number of visitors in Variant A (control) and Variant B (test)
  • Number of conversions in each variant
  • Your chosen confidence threshold — conventionally 95% (p ≤ 0.05), though some teams use 99% for high-stakes decisions
  • Ideally, a predetermined sample size or test duration decided before the test launched

Steps

Step 1: Calculate the conversion rate for each variant

Conversion Rate = Conversions / Visitors

Variant Visitors Conversions Conversion Rate
A (control) 5,000 250 5.0%
B (new design) 5,000 290 5.8%

Step 2: Calculate the relative uplift

Relative Uplift = (Rate B − Rate A) / Rate A = (5.8% − 5.0%) / 5.0% = 16%

This tells you the size of the apparent effect, but not yet whether it's statistically reliable.

Step 3: Calculate the pooled conversion rate

Pooled Rate = (Conversions A + Conversions B) / (Visitors A + Visitors B) = (250 + 290) / (5,000 + 5,000) = 5.4%

Step 4: Calculate the standard error

Standard Error = √[Pooled Rate × (1 − Pooled Rate) × (1/Visitors A + 1/Visitors B)]

Plugging in the numbers: √[0.054 × 0.946 × (1/5,000 + 1/5,000)] ≈ 0.00321

Step 5: Calculate the z-score

Z-Score = (Rate A − Rate B) / Standard Error = (0.05 − 0.058) / 0.00321 ≈ −1.87 (magnitude ≈ 1.87)

Step 6: Convert the z-score to a p-value and compare to your threshold

A z-score of approximately 1.87 corresponds to a two-tailed p-value of roughly 0.061 — above the conventional 0.05 threshold. Despite the visible 16% relative uplift, this result would not be declared statistically significant at the standard 95% confidence level; the test would need a larger sample or longer run to confirm whether the effect is real.

Use the A/B Test Significance calculator to run this calculation directly on your own test data without doing the arithmetic by hand.

Step 7: Decide whether to continue, conclude, or extend the test

If the result is significant, you can confidently roll out the winning variant. If it's close but not yet significant (as in the example above), consider continuing the test to gather more data — provided you decided on a stopping point in advance — rather than stopping as soon as the result looks favourable.

Common Mistakes to Avoid

  • Peeking and stopping as soon as it looks significant — checking results repeatedly and stopping at the first significant-looking moment inflates the false positive rate well above your nominal 5%.
  • Ignoring sample size requirements before launching the test — running an underpowered test that can never reliably detect the effect size you care about wastes traffic and time.
  • Confusing statistical significance with practical significance — a tiny, statistically significant effect on a huge sample may not be worth shipping; a large but not-yet-significant effect may simply need more data.
  • Testing many segments and treating any single significant cut as conclusive — segment-level "wins" found by chance among many comparisons should be treated as hypotheses for a follow-up test, not final results.

Formula & Methodology

Pooled Rate = (Conversions A + Conversions B) / (Visitors A + Visitors B)

Standard Error = √[Pooled Rate × (1 − Pooled Rate) × (1/Visitors A + 1/Visitors B)]

Z-Score = (Rate A − Rate B) / Standard Error

The z-score is converted to a p-value using the cumulative standard normal distribution. This two-proportion z-test assumes a reasonably large sample size (typically at least a few hundred conversions per variant) for the normal approximation to hold reliably; for very small samples, a different statistical test (such as Fisher's exact test) may be more appropriate.

Key Terms

  • Statistical Significance — a measure of confidence that an observed difference is real rather than due to chance
  • Conversion Rate — the percentage of visitors who complete a desired action
  • Standard Deviation — a measure of data variability used in calculating the standard error for significance testing
  • ROAS — Return on Ad Spend; often the downstream business metric an A/B test is ultimately trying to improve

Frequently Asked Questions

A p-value is the probability of observing a difference as large as the one measured (or larger) purely by chance, if there were truly no difference between variants. The conventional threshold for declaring significance is p ≤ 0.05, corresponding to a 95% confidence level — but this is a convention, not a law of statistics, and some teams use stricter thresholds like p ≤ 0.01 for high-stakes decisions.
It depends on your baseline conversion rate and the size of the effect you're trying to detect — smaller baseline rates and smaller effects both require larger samples. Calculate the required sample size before launching the test using your baseline rate and minimum detectable effect, rather than checking significance opportunistically as traffic accumulates, which inflates the chance of a false positive.
No — this practice, known as 'peeking,' dramatically increases the false positive rate, because random fluctuations will cross the 95% significance threshold at some point during many tests purely by chance if you check repeatedly. Decide on a sample size or fixed test duration in advance and only evaluate significance once that predetermined point is reached.
Either continue running the test to collect a larger sample (significance often emerges as sample size grows, assuming the underlying effect is real), or conclude that the observed difference may be due to chance and the test was inconclusive. Avoid the temptation to round up an almost-significant result and ship it as a 'win' — this is one of the most common sources of false confidence in A/B testing programs.
Statistical significance tells you whether an effect is likely real; practical significance tells you whether it's large enough to matter for the business. A 0.1% lift in [conversion rate](/conversion-rate-calculator/) can be statistically significant with a huge sample but not worth shipping, while a meaningful 20% lift can fail to reach significance with too small a sample. Evaluate both the p-value and the absolute size of the effect before making a decision.
First calculate the pooled conversion rate across both groups, then the standard error using that pooled rate and both sample sizes, then divide the difference in conversion rates by the standard error: Z = (Rate A − Rate B) / Standard Error. The resulting z-score is converted to a p-value using the standard normal distribution — the [z-score calculator](/z-score-calculator/) and [A/B test significance calculator](/ab-test-significance-calculator/) both automate this.
Be cautious — testing the same overall result across many segments (device type, country, traffic source) multiplies the chance that at least one segment shows 'significant' results purely by chance. A single significant finding among twenty segment cuts is exactly what random noise would produce. Treat segment-level findings as hypotheses to validate in a dedicated follow-up test, not as conclusive results on their own.
A Type I error (false positive) means concluding there's a real difference when there isn't one — your significance threshold directly controls this rate (a 5% threshold caps Type I errors at 5%). A Type II error (false negative) means missing a real effect that does exist, usually due to an underpowered test with too small a sample. Reducing one error type without increasing sample size typically increases the other.
Yes. Moving from a 95% to a 99% confidence requirement substantially increases the sample size needed to detect the same effect size, because you're demanding stronger evidence before declaring significance. This is a direct trade-off: higher confidence reduces false positives but requires more traffic or a longer test duration to achieve the same statistical power.
Novelty effects are a common culprit — a new design sometimes outperforms during the test partly because it's new and draws extra attention, and that lift can fade once users acclimate to it. Running tests for a sufficient duration (covering multiple full weekly cycles to account for day-of-week effects) and monitoring post-launch performance helps confirm a win holds up over time rather than just during the novelty window.

Related Articles

BEST OF

Best Startup Metric Calculators 2026 — Free Tools for Founders

GUIDE

Startup Metrics Guide — CAC, CLV, Churn & More

GUIDE

Understanding Statistical Significance: Confidence Intervals, Sample Size & Hypothesis Testing

HOW TO

How to Calculate Standard Deviation

HOW TO

How to Calculate Z-Score