A/B Test Significance Calculator
MarketingCheck if your A/B test results are statistically significant. Enter visitors and conversions for both variants to get conversion rates, uplift, p-value, and verdict.
5,000
250
5,000
300
P-Value
0.028
Variant A Conversion Rate
5.00%
Variant B Conversion Rate
6.00%
Relative Uplift (B vs A)
20.00%
Z-Score
2.193
Confidence Level
97.17%
What is a A/B Test Significance?
An A/B Test Significance Calculator determines whether the difference in conversion rates between two test variants โ commonly called Variant A (control) and Variant B (treatment) โ is statistically meaningful or could simply be due to random chance. Marketers, product managers, and growth teams run countless A/B tests on landing pages, email subject lines, and checkout flows, but eyeballing two conversion rates side by side doesn't tell you whether the difference is real or just noise from a limited sample.
This calculator runs a two-proportion z-test, the standard statistical method for comparing two conversion rates, and returns a p-value, z-score, and plain-language verdict on significance. It pairs well with the Conversion Rate Calculator for single-variant reporting and the Sales Funnel Calculator for understanding where in the funnel a tested change has its biggest impact.
How to use this A/B Test Significance calculator
- Enter Variant A โ Visitors, the total number of visitors or sessions exposed to your control variant.
- Enter Variant A โ Conversions, the number of those visitors who completed your goal action.
- Enter Variant B โ Visitors, the total number of visitors exposed to your test variant.
- Enter Variant B โ Conversions, the number of conversions recorded for Variant B.
- Review the P-Value result card โ a value below 0.05 generally indicates a statistically significant difference between variants.
- Check Relative Uplift and Confidence Level together to judge both the size and the certainty of the effect before deciding whether to roll out Variant B.
Formula & Methodology
This calculator uses the two-proportion z-test, the standard statistical method for comparing two conversion rates. Conversion rates: p_A = Conversions_A รท Visitors_A p_B = Conversions_B รท Visitors_B Pooled proportion (assumes no real difference under the null hypothesis): p_pooled = (Conversions_A + Conversions_B) รท (Visitors_A + Visitors_B) Standard error: SE = โ[ p_pooled ร (1 โ p_pooled) ร (1/Visitors_A + 1/Visitors_B) ] Z-score: z = (p_B โ p_A) รท SE P-value (two-tailed, from the standard normal distribution): p-value = 2 ร (1 โ ฮฆ(|z|)) where ฮฆ is the cumulative distribution function of the standard normal distribution. Worked example: Variant A has 5,000 visitors and 250 conversions (5% rate); Variant B has 5,000 visitors and 300 conversions (6% rate). p_pooled = (250 + 300) รท (5,000 + 5,000) = 0.055 SE = โ[0.055 ร 0.945 ร (1/5,000 + 1/5,000)] โ 0.00322 z = (0.06 โ 0.05) รท 0.00322 โ 3.10 p-value โ 0.0019 โ statistically significant at well above 95% confidence
Frequently Asked Questions
Statistical significance means the difference observed between two variants is unlikely to have happened by random chance alone, typically judged against a 95% confidence threshold (a p-value below 0.05). If a result is significant, you can be reasonably confident that the better-performing variant is actually better, not just lucky in this particular sample.
The p-value is the probability of observing a difference as large as (or larger than) the one in your data, assuming there is actually no real difference between the variants. A low p-value (typically below 0.05) suggests the observed difference is unlikely to be due to chance, while a high p-value means the data doesn't provide strong evidence of a real effect.
The two-proportion z-test calculates a pooled conversion rate across both variants, then computes a standard error based on that pooled rate and both sample sizes. The z-score is the difference between the two variants' conversion rates divided by this standard error, and the p-value is derived from where that z-score falls on the standard normal distribution.
Statistical significance tells you whether a difference is likely real rather than random noise, while practical significance asks whether that difference is large enough to matter for your business. A test can be statistically significant with a tiny 0.3% uplift that isn't worth implementing, so always look at the [Relative Uplift](/conversion-rate-calculator/) figure alongside the p-value.
There's no fixed minimum, but checking significance with very small sample sizes (under a few hundred visitors per variant) tends to produce unreliable, noisy results that can flip with the next few conversions. Most practitioners wait until each variant has at least a few hundred conversions before drawing conclusions, and ideally pre-determine a sample size target before starting the test.
Stopping a test the moment it crosses the significance threshold is a common mistake known as 'peeking,' which inflates the false-positive rate because random fluctuations can briefly cross the threshold before settling back. It's safer to decide your sample size or test duration in advance and only check significance once that predetermined point is reached.
A negative relative uplift means Variant B converted at a lower rate than Variant A, so if the test is significant, it indicates Variant A is the better performer rather than B. Always check the sign of the uplift alongside the p-value, since a significant result with negative uplift means you should keep or revert to the original variant.
Conversion rate for each variant is simply conversions divided by visitors, expressed as a percentage โ for example, 250 conversions from 5,000 visitors gives a 5% conversion rate. This calculator computes this rate independently for Variant A and Variant B before running the significance test on the difference between them.
The required sample size depends on your baseline conversion rate, the minimum uplift you want to reliably detect, and your desired confidence level โ smaller expected effects require larger samples to detect reliably. As a practical guide, most marketing A/B tests run with at least a few thousand visitors per variant to reach a meaningful, stable conclusion.
This usually happens when a test was stopped too early or didn't run long enough to capture day-of-week or seasonal variation in visitor behaviour, which can temporarily skew conversion rates in one direction. Running tests for at least one to two full business cycles (often one to two weeks) helps avoid drawing conclusions from a temporary anomaly.
A 90% confidence level (p-value below 0.10) is sometimes used for lower-stakes decisions or early-stage tests where faster iteration matters more than rigour, but 95% confidence is the standard threshold for most marketing and product decisions. Choose your threshold before running the test based on how costly a wrong decision would be.
The [Conversion Rate Calculator](/conversion-rate-calculator/) computes a single variant's conversion rate from visitors and conversions, while this A/B Test Significance Calculator compares two variants against each other to determine whether their difference is statistically meaningful. Use the Conversion Rate Calculator for single-funnel reporting and this calculator specifically when comparing test variants.
Also known as