A/B test significance calculator
Enter visitors and conversions for each variant to see the p-value, lift, and a plain-English verdict - or switch to planner mode to size a test before you start.
A/B test significance calculator
Find out if your test result is real - or plan how big a test you need before you start.
Variant A (control)
Variant B (challenger)
Enter both variants to see the verdict.
Quick answer: Statistical significance tells you whether the difference between two variants is real or just random noise. This calculator runs a two-proportion z-test on your visitors and conversions, returns a p-value, and gives a clear verdict at your chosen confidence level. If the p-value is at or below your significance threshold (0.05 at 95% confidence), the result is significant.
How the z-test works
For each variant we compute the conversion rate (conversions / visitors). We then calculate a pooled standard error and a z-score for the difference between the two rates. The z-score is converted to a p-value using the normal distribution. A smaller p-value means the observed difference is less likely to be due to chance.
Confidence levels and z-thresholds
| Confidence | Significance (alpha) | Two-tailed z-threshold | When to use |
|---|---|---|---|
| 90% | 0.10 | 1.645 | Low-risk, exploratory tests |
| 95% | 0.05 | 1.960 | Standard for marketing & product |
| 99% | 0.01 | 2.576 | High-stakes decisions |
Plan a test before you run it
Switch to planner mode to find the sample size you need. Enter your baseline conversion rate and the minimum detectable effect (the smallest relative lift worth catching). The calculator returns the required visitors per variant and, with your daily traffic, an estimated test duration. Deciding sample size upfront is the single best defense against false positives.
How to use this calculator
- Enter your data - visitors and conversions for control and variant.
- Read the verdict - p-value, lift, confidence interval, and a clear significant/not-significant call.
- If not significant, keep running to your pre-planned sample size instead of stopping early.
- Plan the next test with planner mode - enter baseline rate and minimum detectable effect to get the required sample size.
Common A/B testing mistakes
- Peeking and stopping early. Calling a winner the moment you see p < 0.05 dramatically inflates false positives - set the sample size first.
- Testing too small a sample. Underpowered tests can't detect realistic lifts; use the planner.
- Running too many variants at once without correcting for multiple comparisons.
- Ignoring practical significance. A statistically significant 0.1% lift may not be worth shipping.
- Changing the test mid-flight - altering the page or traffic split invalidates the result.
Frequently asked questions
- What is statistical significance in an A/B test?
- It is the probability that the difference between control and variant is real rather than random chance. This tool runs a two-proportion z-test and compares the p-value against your significance level.
- How many visitors do I need?
- It depends on your baseline rate and the smallest lift you want to detect - smaller effects need much larger samples. Use planner mode to get the exact number per variant.
- What is a p-value?
- The probability of seeing a difference at least as large as observed if there were truly no difference. Below 0.05 suggests the result is unlikely to be chance.
- Why shouldn't I stop a test early?
- Peeking and stopping at the first sign of significance inflates false positives. Set your sample size first, then run to completion before deciding.
- What confidence level should I use?
- 95% is standard. Use 99% for costly decisions and 90% only for low-risk exploratory tests.
- One-tailed vs two-tailed test?
- This tool uses a two-tailed test, which detects a variant that's better or worse. One-tailed only checks for improvement and is faster but can miss a variant that's hurting you - two-tailed is the safer default.
- What is statistical power?
- The probability of detecting a real effect when one exists, usually set to 80%. Low power means missing genuine winners; the planner sizes tests for adequate power.
- Statistical vs practical significance?
- Statistical significance means a result is unlikely to be random; practical significance asks if the lift is big enough to matter. A tiny lift can be significant yet not worth shipping.
- Is this A/B test calculator free?
- Yes - free, no signup, and entirely in your browser. Use it to evaluate finished tests and to plan sample size for new ones.
Related tools
Related free tools
Keep going with these tools
Landing Page Grader
Grade your landing page in 10 seconds
Conversion Rate Benchmarker
Compare your rate against your industry
SaaS Pricing Page Grader
Grade your pricing page in 2 minutes
Website Worth Calculator
What is your website worth?
SaaS Valuation Calculator
What is your SaaS worth?
App Idea Validator
Score your app idea in 60 seconds
Want a team to handle it for you? We turn validated ideas into shippable MVPs in weeks, not months. Explore AI MVP development.
Related guides
Go deeper with our guides
Practical, up-to-date guides on the strategy, costs, and execution behind this tool.
A/B Testing for Small Business Websites: What to Test First in 2026
What to test first, tools for small budgets, sample sizes, common mistakes, and when not to test.
Conversion Rate Optimization for Small Business Websites 2026
A practical CRO guide: find conversion leaks, fix forms and CTAs, add trust signals, and measure results.
Landing Page Design 2026: Beat the 6.6% Median Conversion Rate
Design principles, psychological triggers, and testing frameworks that turn landing pages into conversion engines.
Ready to ship your next product?
Tell us what you're building. Senior engineers will scope, plan, and start delivering your product with production-ready architecture - fast.