What is sample size?
Sample size is the number of people, visits, or events included in a test or study. It needs to be large enough to detect the effect you care about with reasonable confidence.
Last reviewed .
What determines the sample size you need
Four things drive it: your baseline conversion rate, the smallest improvement you want to detect, the confidence level, and the statistical power you want. A smaller expected effect needs a larger sample.
Online calculators and testing tools can estimate the number before you start.
Problems with small samples
A small sample can produce striking results that vanish when repeated, because chance plays a bigger role. It can also miss real effects, giving a false sense that nothing changed.
Decide the sample size before the test begins, and avoid stopping early because early results look good. Run the test for whole weeks to cover natural cycles in behavior.
Common questions
How do you calculate sample size for an A/B test?
Use a sample size calculator with your baseline conversion rate, the minimum effect you want to detect, your confidence level, and your power. The tool returns the required number per variant.
What is statistical power?
Power is the chance that a test detects a real effect of a given size. Many teams aim for 80%, though the right level depends on the stakes.
Can you run a test with low traffic?
You can, but you'll need a longer test or a bigger expected effect. For very low traffic, testing bold changes works better than small tweaks.
Is more data always better?
More data gives more precise estimates, but it also takes more time and may expose more people to a weaker option. Aim for enough, not unlimited.
Want help putting this to work?
Tell us where your marketing stands today, and we'll show you what to do first.