Skip to content

A/B Test Calculator

Enter your test results to see whether the difference between your control and each variant is statistically significant.

A (control)
Variant B
Statistically significantTests use a 95% confidence level.A (control)Conversion rate: 5.00%Variant B: significant vs AConversion rate: 5.80%Change: +16.00%Confidence: 98.76%Z-score: 2.503P-value: 0.0123Range: +3.47% to +28.53%Confidence here is 1 minus the p-value. It isn't the chance that a variant is better.
Change from A
  • B: measured +16.00%. At 95% confidence, the range consistent with the test is +3.47% to +28.53%. The whole range is above 0%, so B tested better than A.

The dot is the change you measured. The bar is the range consistent with the significance test, shown as an approximate relative change. A larger planned sample narrows the bar. Green bars are significantly better than A, orange bars are significantly worse, and gray bars cross 0% and aren't significant.

How the calculator works

The calculator uses a pooled two-proportion z-test. It estimates how likely a difference at least this large would be if the groups truly performed the same.

With several variants, it compares each one to the control and applies a Bonferroni correction, and a chi-square test checks all groups together. Confidence is 1 minus the p-value, adjusted when you test several variants.

How to run a cleaner test

Plan your sample size before you start with the Sample Size Calculator, and read the result once you reach it. Checking repeatedly and stopping at the first significant reading raises your chance of a false win.

Significance tells you a difference is unlikely to be chance. It doesn't tell you the difference is large enough to matter, so weigh the size of the lift as well.

Disclaimer

This tool is for informational purposes only, and we don't guarantee the accuracy of its results. You're responsible for verifying any results before relying on them. See our terms of use.

Common questions

Is my information saved?

No. Your numbers stay in your browser, and they aren't sent to anyone.

Is there a Bayesian version of this calculator?

Yes. It shows the probability, under its model, that one version beats another, instead of a p-value.

Try the Bayesian A/B Test Calculator

How many visitors does my test need?

Work it out before the test starts, based on your current conversion rate and the smallest lift that would matter to you.

Try the Sample Size Calculator

What does the range in the chart show?

It's the range of changes consistent with the significance test, using the same margin of error and confidence level, so it never contradicts the verdict. It's shown as a relative change, which is an approximation, and it isn't the same as a textbook confidence interval.

What does confidence mean in the results?

It's 1 minus the p-value. A higher number means a difference at least this large would be less likely if the groups truly performed the same. It isn't the probability that a variant is better.

What confidence level should I use?

95% is a common choice and this calculator's default. A higher level lowers the risk of a false win but needs more visitors. Choose it before the test starts.

Can I test more than two versions?

Yes. You can compare up to four variants against a control. Each extra comparison raises the chance of a false win, so the calculator applies a Bonferroni correction, and your test needs more visitors.

Should I use a one-tailed or two-tailed test?

Use two-tailed in most cases, because it detects a change in either direction. A one-tailed test only asks whether a variant beat the control, so choose it before the test starts, and only if a decline wouldn't change your decision.

Related terms

See our Conversion Rate Optimization servicesSee all tools

Want help putting this to work?

Tell us where your marketing stands today, and we'll show you what to do first.

Talk to an expertCall (800) 935-5133