What is A/B testing?
A/B testing is a method of comparing two versions of the same page, ad, or email by showing each to a different, randomly assigned group of people. You then measure which version produces more of the result you want, such as sign-ups or purchases.
Last reviewed .
How an A/B test works
You start with a control, which is your current version, and a variation that changes one thing, such as a headline or button. Visitors are split at random between the two, so each group is similar and the difference in results can be traced to the change.
You decide in advance which metric decides the winner. Picking it before the test starts keeps you from choosing whichever number looks best afterward.
What to test and what to avoid
Good candidates are the elements closest to the decision: headlines, calls to action, form length, page layout, and offers. Changes to these tend to affect behavior more than small cosmetic tweaks.
Avoid stopping a test the moment one version pulls ahead. Early gaps often shrink, so let the test reach the sample size and duration you planned before you call a winner.
How A/B testing differs from multivariate testing
An A/B test compares whole versions, or one change at a time. Multivariate testing changes several elements at once to learn how they work together, and it needs much more traffic to give a clear answer.
Common questions
How long should an A/B test run?
Run it until you reach the sample size you planned, and long enough to cover a full cycle of your normal traffic, such as whole weeks. Stopping early raises the chance of picking a false winner.
How much traffic do you need for an A/B test?
It depends on your current conversion rate and the size of the lift you want to detect. Smaller expected changes need more visitors. A sample size calculator can estimate the number before you start.
What is the difference between A/B testing and split testing?
The terms are often used interchangeably. Some people use split testing for tests that send traffic to entirely different URLs, and A/B testing for changes made within one page.
What if an A/B test has no clear winner?
An inconclusive result means the test did not establish a clear difference at the planned precision. Check the confidence interval and minimum detectable effect: insufficient data can hide a meaningful effect. Only conclude the impact is negligible when the interval or an equivalence analysis supports that conclusion.
Want help putting this to work?
Tell us where your marketing stands today, and we'll show you what to do first.