What is a holdout test?
A holdout test deliberately withholds a campaign from a randomly chosen part of your audience, called the holdout group. Comparing it with the group that saw the campaign shows how much extra the campaign produced.
Last reviewed .
How a holdout test works
You randomly split your audience. Most people receive the campaign, and a small share doesn't. After the test, you compare conversion rates between the groups, and the difference is the campaign's incremental effect.
Holdouts are used for email, remarketing, push notifications, and other channels where you control who is eligible.
Designing a reliable holdout
Randomize assignment, and keep the holdout group clear of the campaign for the whole test. If it's too small, results will be noisy, and if it's too large, you give up conversions you could have won.
Run it long enough to cover a normal buying cycle, and check statistical significance before you act.
Common questions
What is the difference between a holdout and a control group?
A holdout is a control group made by withholding a campaign from part of the audience. Control group is the broader term for any comparison group.
How big should a holdout group be?
Big enough to detect the effect you expect. Smaller expected effects need bigger groups, and a sample size calculator can help you choose.
Does a holdout cost me sales?
It may, because the holdout group doesn't see the ads. Many teams accept this small cost for reliable proof of what the campaign adds.
Where are holdout tests commonly used?
They're common in email and CRM programs, retargeting, and any channel where you can pick who is eligible to be reached.
Want help putting this to work?
Tell us where your marketing stands today, and we'll show you what to do first.