AB Testing Significance Calculator – Find the Real A/B Test Winner in 3 Easy Steps
A/B Testing Significance Calculator
Use this A/B testing significance calculator to check if your split test results are statistically significant before making marketing decisions.
Use this free A/B testing significance calculator (also called an AB testing significance calculator) to find out whether the difference between two versions of a page, email or ad is real or just chance. Enter visitors and conversions for the control and the variant, pick a confidence level, and the tool shows the conversion rates, uplift, z-score, significance and a clear winner or "keep testing" verdict.
Quick answer
An A/B test is statistically significant when the z-score of the difference between the two conversion rates passes the critical value for your confidence level: 1.645 for 90%, 1.96 for 95% and 2.576 for 99%. With 10,000 visitors each, 250 conversions for A (2.5%) and 310 for B (3.1%) give a z-score of 2.572, so B wins at 95% confidence with a 24% relative uplift.
Table of Contents
A variant that converts better for a few days can still be a fluke. Calling a winner too early is one of the most common A/B testing mistakes, and it leads to changes that do nothing, or even hurt sales, once rolled out.
This AB testing significance calculator applies a standard two-proportion z-test, so you can make the call with numbers instead of gut feeling. It also estimates the extra revenue from the winning version.
What Is Statistical Significance in A/B Testing?
Statistical significance tells you how unlikely your result would be if the two versions actually performed the same. A 95% confidence level means you accept a 5% chance of declaring a winner when there is no real difference. The higher the confidence level, the more data you need.
A/B Test Significance Formula
- Conversion rate: p = conversions ÷ visitors, for A and for B
- Pooled rate: p̂ = (conversions A + conversions B) ÷ (visitors A + visitors B)
- Standard error: SE = √[p̂ × (1 − p̂) × (1 ÷ visitors A + 1 ÷ visitors B)]
- Z-score: z = (pB − pA) ÷ SE
- Relative uplift: (pB − pA) ÷ pA × 100
The tool compares the z-score with the critical value for your chosen level. The significance figure it shows is the one-sided confidence that B is better than A, which is how most marketing teams read test results.
Critical Z-Values by Confidence Level
| Confidence level | Critical z-value | Use when |
|---|---|---|
| 90% | 1.645 | Low-risk changes, fast decisions |
| 95% | 1.960 | Standard choice for most tests |
| 99% | 2.576 | High-risk changes such as pricing |
How to Use the AB Testing Significance Calculator
Step 1: Enter Visitors and Conversions
Choose a confidence level, your currency and the average order or goal value. Then enter total visitors and conversions for Group A (the original) and Group B (the variant). The conversion rate for each group appears straight away.
Step 2: Read the Winner and Significance
The result panel updates as you type. Here B converts at 3.10% against 2.50% for A, a 24% uplift. The z-score of 2.572 passes the 1.96 threshold, so B is the winner, and the extra revenue at a $50 order value is about $3,000 per 10,000 visitors.
Step 3: Know When to Keep Testing
If B had 270 conversions instead of 310, the uplift falls to 8% and the z-score to 0.889. That is well below 1.96, so the tool says there is no significant difference yet. Keep the test running or test a bolder change.
A/B Test Examples
| Visitors per group | Conv. A | Conv. B | Z-score | Significant at 95%? |
|---|---|---|---|---|
| 10,000 | 250 | 310 | 2.572 | Yes |
| 10,000 | 250 | 270 | 0.889 | No |
| 2,000 | 40 | 60 | 2.025 | Yes, just |
| 50,000 | 1,000 | 1,100 | 2.205 | Yes |
A/B Testing Best Practices
- Decide your sample size and test length before you start, and do not stop the moment the result looks good.
- Run tests for at least one or two full weeks to cover weekday and weekend behaviour.
- Split traffic randomly and evenly between versions.
- Test one main change at a time so you know what caused the difference.
- Watch revenue per visitor too, not just conversion rate.
Once you have a winner, estimate the impact on monthly revenue with the CRO Calculator. For tests with more than two categories, use the Chi-Square Calculator.
Frequently Asked Questions
What does an A/B testing significance calculator do?
It checks whether the difference in conversion rate between two versions is large enough, given the sample size, to be unlikely to be random chance.
What confidence level should I use?
95% is the standard for most marketing tests. Use 90% for low-risk changes where speed matters, and 99% for high-stakes changes such as pricing.
How many visitors do I need for an A/B test?
It depends on your conversion rate and the size of the change. Small uplifts on low conversion rates can need tens of thousands of visitors per version.
What is a z-score in A/B testing?
The z-score measures how many standard errors apart the two conversion rates are. A z-score above 1.96 means the result is significant at 95% confidence.
Why does the tool say no significant difference when B is higher?
Because the gap is too small for the amount of data. Random variation alone could produce it, so keep testing.
Is my data stored?
No. The calculator runs entirely in your browser.