Home›Calculators›Marketing›A/B Test Significance Calculator

A/B Test Significance Calculator

Marketing

Check if your A/B test results are statistically significant. Enter visitors and conversions for both variants to get conversion rates and verdict.

Reviewed by the thecalcu.com team · Last updated June 30, 2026

5,000
250
5,000
300

P-Value

0.028
Variant A Conversion Rate
5.00%
Variant B Conversion Rate
6.00%
Relative Uplift (B vs A)
20.00%
Z-Score
2.193
Confidence Level
97.17%

This calculator computes your P-Value, Variant A Conversion Rate, Variant B Conversion Rate, Relative Uplift (B vs A), Z-Score, Confidence Level from the values you enter.

Inputs
Variant A — VisitorsVariant A — ConversionsVariant B — VisitorsVariant B — Conversions
Outputs
P-ValueVariant A Conversion RateVariant B Conversion RateRelative Uplift (B vs A)Z-ScoreConfidence Level

What is a A/B Test Significance?

An A/B Test Significance Calculator determines whether the difference in conversion rates between two test variants, commonly called Variant A (control) and Variant B (treatment), is statistically meaningful or could simply be due to random chance. Marketers, product managers, and growth teams run countless A/B tests on landing pages, email subject lines, and checkout flows, but eyeballing two conversion rates side by side doesn't tell you whether the difference is real or just noise from a limited sample.

This calculator runs a two-proportion z-test, the standard statistical method for comparing two conversion rates, and returns a p-value, z-score, and plain-language verdict on significance. It pairs well with the Conversion Rate Calculator for single-variant reporting and the Sales Funnel Calculator for understanding where in the funnel a tested change has its biggest impact.

Why Use an A/B Test Significance Calculator?

Many teams make the costly mistake of declaring an A/B test "won" based on a small difference in raw conversion numbers, without checking whether that difference is statistically robust. Implementing a "winning" variant that was actually just random noise can mean rolling out a change that provides no real benefit, or worse, reverting a genuinely better-performing control.

This calculator removes the guesswork by running the actual statistical test marketers and data scientists use to validate test results. A team deciding whether to roll out a new checkout button color, or a growth marketer comparing two landing page headlines, gets a clear p-value and verdict instead of relying on intuition about which numbers "look" different enough.

Who Should Use This Calculator?

  • Growth marketers running landing page, email, or ad creative A/B tests who need to confirm a result before rolling it out.
  • Product managers evaluating feature experiments or UI changes where conversion or sign-up rate is the success metric.
  • Conversion rate optimisation (CRO) specialists who run continuous testing programmes and need a quick significance check between tests.
  • E-commerce teams testing checkout flow changes, where even small uplifts matter but false positives are costly to implement.
  • Data analysts who want a fast sanity check on test results before running a more detailed analysis, alongside tools like the Z-Score Calculator.

What Insights Does the A/B Test Significance Calculator Give You?

P-Value is the headline output, it tells you the probability that the observed difference between variants happened by chance alone; a value below 0.05 is the conventional threshold for declaring significance. Variant A Conversion Rate and Variant B Conversion Rate show each variant's raw performance independently, letting you see the actual numbers behind the test.

Relative Uplift (B vs A) expresses how much better (or worse) Variant B performed compared to A as a percentage, this tells you the practical size of the effect, separate from whether it's statistically significant. Z-Score is the underlying test statistic used to derive the p-value, useful if you want to cross-reference against a standard z-table. Confidence Level restates the result as a percentage confidence (100% minus the p-value as a percentage), making the significance threshold easier to read at a glance, 95% or higher generally indicates a significant result.

How to use this A/B Test Significance calculator

  1. Enter Variant A, Visitors, the total number of visitors or sessions exposed to your control variant.
  2. Enter Variant A, Conversions, the number of those visitors who completed your goal action.
  3. Enter Variant B, Visitors, the total number of visitors exposed to your test variant.
  4. Enter Variant B, Conversions, the number of conversions recorded for Variant B.
  5. Review the P-Value result card, a value below 0.05 generally indicates a statistically significant difference between variants.
  6. Check Relative Uplift and Confidence Level together to judge both the size and the certainty of the effect before deciding whether to roll out Variant B.
Show formula & methodology ↓Show less ↑

Formula & Methodology

This calculator uses the two-proportion z-test, the standard statistical method for comparing two conversion rates.

Conversion rates:

p_A = Conversions_A ÷ Visitors_A
p_B = Conversions_B ÷ Visitors_B

Pooled proportion (assumes no real difference under the null hypothesis):

p_pooled = (Conversions_A + Conversions_B) ÷ (Visitors_A + Visitors_B)

Standard error:

SE = √[ p_pooled × (1 − p_pooled) × (1/Visitors_A + 1/Visitors_B) ]

Z-score:

z = (p_B − p_A) ÷ SE

P-value (two-tailed, from the standard normal distribution):

p-value = 2 × (1 − Φ(|z|))

where Φ is the cumulative distribution function of the standard normal distribution.

Worked example: Variant A has 5,000 visitors and 250 conversions (5% rate); Variant B has 5,000 visitors and 300 conversions (6% rate).

p_pooled = (250 + 300) ÷ (5,000 + 5,000) = 0.055
SE = √[0.055 × 0.945 × (1/5,000 + 1/5,000)] ≈ 0.00322
z = (0.06 − 0.05) ÷ 0.00322 ≈ 3.10
p-value ≈ 0.0019 → statistically significant at well above 95% confidence

Frequently Asked Questions

What is statistical significance in A/B testing?
Statistical significance means the difference observed between two variants is unlikely to have happened by random chance alone, typically judged against a 95% confidence threshold (a p-value below 0.05). If a result is significant, you can be reasonably confident that the better-performing variant is actually better, not just lucky in this particular sample.
What is a p-value and how do I interpret it?
The p-value is the probability of observing a difference as large as (or larger than) the one in your data, assuming there is actually no real difference between the variants. A low p-value (typically below 0.05) suggests the observed difference is unlikely to be due to chance, while a high p-value means the data doesn't provide strong evidence of a real effect.
What is the formula for a two-proportion z-test?
The two-proportion z-test calculates a pooled conversion rate across both variants, then computes a standard error based on that pooled rate and both sample sizes. The z-score is the difference between the two variants' conversion rates divided by this standard error, and the p-value is derived from where that z-score falls on the standard normal distribution.
What is the difference between statistical significance and practical significance?
Statistical significance tells you whether a difference is likely real rather than random noise, while practical significance asks whether that difference is large enough to matter for your business. A test can be statistically significant with a tiny 0.3% uplift that isn't worth implementing, so always look at the [Relative Uplift](/conversion-rate-calculator/) figure alongside the p-value.
How many visitors do I need before checking significance?
There's no fixed minimum, but checking significance with very small sample sizes (under a few hundred visitors per variant) tends to produce unreliable, noisy results that can flip with the next few conversions. Most practitioners wait until each variant has at least a few hundred conversions before drawing conclusions, and ideally pre-determine a sample size target before starting the test.
Can I stop my A/B test as soon as it becomes significant?
Stopping a test the moment it crosses the significance threshold is a common mistake known as 'peeking,' which inflates the false-positive rate because random fluctuations can briefly cross the threshold before settling back. It's safer to decide your sample size or test duration in advance and only check significance once that predetermined point is reached.
What does a negative relative uplift mean?
A negative relative uplift means Variant B converted at a lower rate than Variant A, so if the test is significant, it indicates Variant A is the better performer rather than B. Always check the sign of the uplift alongside the p-value, since a significant result with negative uplift means you should keep or revert to the original variant.
How is conversion rate calculated for each variant?
Conversion rate for each variant is simply conversions divided by visitors, expressed as a percentage, for example, 250 conversions from 5,000 visitors gives a 5% conversion rate. This calculator computes this rate independently for Variant A and Variant B before running the significance test on the difference between them.
What sample size should I use for an A/B test?
The required sample size depends on your baseline conversion rate, the minimum uplift you want to reliably detect, and your desired confidence level, smaller expected effects require larger samples to detect reliably. As a practical guide, most marketing A/B tests run with at least a few thousand visitors per variant to reach a meaningful, stable conclusion.
Why did my test show significance but the result reversed later?
This usually happens when a test was stopped too early or didn't run long enough to capture day-of-week or seasonal variation in visitor behaviour, which can temporarily skew conversion rates in one direction. Running tests for at least one to two full business cycles (often one to two weeks) helps avoid drawing conclusions from a temporary anomaly.
Is a 90% confidence level acceptable instead of 95%?
A 90% confidence level (p-value below 0.10) is sometimes used for lower-stakes decisions or early-stage tests where faster iteration matters more than rigour, but 95% confidence is the standard threshold for most marketing and product decisions. Choose your threshold before running the test based on how costly a wrong decision would be.
How does this differ from the Conversion Rate Calculator?
The [Conversion Rate Calculator](/conversion-rate-calculator/) computes a single variant's conversion rate from visitors and conversions, while this A/B Test Significance Calculator compares two variants against each other to determine whether their difference is statistically meaningful. Use the Conversion Rate Calculator for single-funnel reporting and this calculator specifically when comparing test variants.
Also known as
ab testing calculatorsplit test significance calculatortwo-proportion z-test calculatorconversion rate significance teststatistical significance calculator