Concluding an A/B test
Decide whether the observed difference is significant or could be chance.
← All free tools
Free tool · A/B Test
A/B Test Calculator | significance
Check whether the difference between your control and your variant is statistically significant. Two-proportion z-test, p-value, z-score, uplift and confidence interval.
Control (A)
Rate
4.20%
Variant (B)
Rate
4.80%
Significant result
The observed difference is significant at the 5% threshold.
p-value
0.0407
z-score
2.047
Uplift
+14.29%
Variant CI
4.38% – 5.22%
Two-sided (or one-sided) z-test on two independent proportions. Not a substitute for in-depth analysis or a Bayesian approach for long or multi-variant tests.
How it works
A z-test on two proportions, in 4 steps
The calculator compares two observed conversion rates and computes the probability that the difference is due to chance.
01
Enter the control (A)
Visitors and conversions observed in the control group over the test period.
02
Enter the variant (B)
Visitors and conversions in the variant group exposed to the change under test.
03
Pick your alpha
5% is the standard. 1% for critical decisions. 10% for iterative testing.
04
Read the verdict
If p-value < alpha, the difference is statistically significant. Otherwise, keep collecting.
Formula
z = (pv − pc) / √(p̄ · (1−p̄) · (1/nc + 1/nv))
Where pc, pv are the observed rates, p̄ the pooled rate, nc and nv the visitor counts. p-value = 2 · (1 − Φ(|z|)) two-sided. Variant CI: pv ± Zα/2 · √(pv(1−pv)/nv).
When to use it
4 typical use cases
Monthly CRO reporting
Document p-value, uplift and CI for every test shipped to the leadership team.
Auditing past tests
Check the statistical robustness of tests run in-house.
Pricing / redesign decisions
Before shipping a variant to production, make sure it actually beats the control.
FAQ
Frequently asked questions
What is statistical significance?
It tells you whether the difference observed between your variants is likely real or could be due to chance. A result is considered significant when the p-value is below the chosen alpha threshold (5% by default).
How do I interpret the p-value?
The p-value is the probability of observing a difference at least as large as the one observed, assuming there is actually no difference. The smaller it is, the more significant the result. A p-value of 0.03 means a 3% chance of a false positive.
Two-sided or one-sided test?
Two-sided is the recommended standard: you want to know whether the variant differs from the control (better or worse). One-sided assumes you only expect an improvement and will not detect a degradation — the risk is underestimating a negative effect.
What does the confidence interval (CI) mean?
The 95% CI gives the range where the variant’s true conversion rate most likely sits. If it does not overlap the control’s rate, that is a strong significance signal.
My result is not significant — what now?
Three options: (1) keep collecting traffic if the planned duration is not reached, (2) accept that the difference is small and stop the test, (3) reformulate a bolder hypothesis to target a larger MDE. Our MDE calculator pairs well with this analysis.
Does the calculator handle multiple variants?
This tool compares 2 groups (A vs B). For 3+ variants, apply a Bonferroni correction (alpha / number of comparisons) and treat each pair separately.
Can I stop the test as soon as it is significant?
No — that is the peeking trap: checking results too early inflates the false positive rate. Define a duration and sample size upfront (with our MDE calculator) and stick to it.
Need a senior opinion on your A/B tests?
Full CRO audit: roadmap, hypotheses, tracking and execution. 5 days, €0 for B2B scale-ups.