Cost-Based Confidence Calculator
Find out whether one version really costs less per conversion, or whether the difference could be down to chance.
B costs significantly less
At 95% confidence.
- Cost per conversion, A
- $50.00
- Cost per conversion, B
- $38.46
- Absolute change
- −$11.54
- Percent change
- −23.08%
- Confidence
- 95.14%
- Z-score
- −1.972
- P-value
- 0.0486
- Range
- −40.73% to −0.17%
Confidence here is 1 minus the p-value. It isn't the chance that B costs less.
- B: measured −23.08%. At 95% confidence, the range consistent with the test is −40.73% to −0.17%. The whole range is below 0%, so B costs less per conversion than A.
The dot is the change you measured. The bar is the range consistent with the significance test. A larger planned sample narrows the bar. Green bars are significantly better than A, orange bars are significantly worse, and gray bars cross 0% and aren't significant.
How the calculator works
Conversions vary from one period to the next, even when nothing changes. So the calculator treats each group's conversions as a Poisson count for the money spent, and it runs a z-test on the log of the two costs per conversion.
The result depends on your conversion counts more than your spend. Two groups with the same cost per conversion can give very different answers if one has 20 conversions and the other has 2,000.
The chart shows the measured change in cost per conversion and the range of changes the test can't rule out. A lower cost is the good result, so a bar entirely below 0% is green.
When to use it
Use it when groups spend different amounts, or when visitor counts aren't the main story, such as comparing two ad sets, bid strategies, or campaigns.
Compare groups from the same period and audience where you can. The test can tell you a difference is unlikely to be chance, but it can't tell you the change caused it unless the groups were split fairly.
Disclaimer
This tool is for informational purposes only, and we don't guarantee the accuracy of its results. You're responsible for verifying any results before relying on them. See our terms of use.
Common questions
Is my information saved?
No. Your numbers stay in your browser, and they aren't sent to anyone.
How many conversions do I need?
There's no fixed number, but small counts are unreliable, so the calculator flags any group with fewer than 10 conversions. If your result isn't significant, more conversions may settle it, or the real difference may be small.
What does the range in the chart show?
It's the range of changes in cost per conversion consistent with the significance test, at the confidence level you chose. It uses the same margin of error as the test, so it never contradicts the verdict. A one-tailed test only limits one side, so its bar runs to the edge.
What does confidence mean in the results?
It's 1 minus the p-value. A higher number means a difference at least this large would be less likely if the two costs per conversion were truly the same. It isn't the probability that B costs less.
What does this test assume?
It treats spend as fixed and only models the randomness in conversions, so it doesn't account for swings in auction prices or the value of each conversion.
Should I use a one-tailed or two-tailed test?
Use two-tailed in most cases, because it detects a change in either direction. A one-tailed test only looks in the direction you choose, so pick it before you look at the results.
Want help putting this to work?
Tell us where your marketing stands today, and we'll show you what to do first.