Bayesian A/B Test Calculator
Find out how likely it is that one version truly beats another, and what you stand to lose if you pick the wrong one.
Chance B beats A
97.9%
B is likely better
That meets your 95% decision threshold.
- Estimated rate for A
- 5.04%
- Estimated rate for B
- 6.54%
- Expected lift
- +30.93%
- 95% credible interval
- +1.14% to +67.42%
- Expected loss if you pick B
- 0.005 points
The chance B beats A is calculated exactly. The credible interval, expected loss, and chart are estimates from 20,000 simulations.
Under this model, there's a 95% chance the true lift is between +1.14% and +67.42%. The whole interval is above 0%, so B is very likely better than A.
Each bar shows how often the simulations landed on that lift, so taller bars are more likely. Green bars are lifts where B beats A, and orange bars are lifts where B loses. The dashed lines mark the 95% credible interval.
How the calculator works
It uses a Beta-Binomial model with a flat Beta(1, 1) prior, which gives every conversion rate equal prior weight for each version. Your visitors and conversions then shape an estimate of each version's true conversion rate.
The chance B beats A is calculated exactly from the two Beta distributions. For the credible interval, expected loss, and chart, it draws 20,000 simulated rates for each version and compares them. The simulation uses a fixed seed, so the same numbers always give the same result.
How to read the results
The credible interval is the range where the true lift likely falls, with 95% probability under this model. If it includes 0%, B could still turn out to be worse.
Expected loss is the conversion rate you'd give up on average if you pick B, measured in percentage points. A small expected loss can justify a decision even when the chance of winning falls short of your threshold.
Disclaimer
This tool is for informational purposes only, and we don't guarantee the accuracy of its results. You're responsible for verifying any results before relying on them. See our terms of use.
Common questions
Is my information saved?
No. Your numbers stay in your browser, and they aren't sent to anyone.
How is this different from a standard A/B test calculator?
A standard calculator gives a p-value, which describes how surprising your data would be if the versions truly performed the same. This one gives the probability that B is better, which is often easier to explain. They can disagree slightly, because they answer different questions.
Try the A/B Test CalculatorWhat decision threshold should I use?
It depends on the cost of being wrong. Use a higher threshold for changes that are expensive or hard to undo, and choose it before the test starts.
Can I stop a test early with this calculator?
Be careful. Early results swing widely on small samples, so plan a sample size first and let the test run through a full business cycle.
Want help putting this to work?
Tell us where your marketing stands today, and we'll show you what to do first.