Holdout group
Holdout group (control group)
Email Marketing
A holdout group is a random slice of an audience that deliberately gets no message, so you can measure how much the message really changed behavior.
What is Holdout group?
An A/B test compares message against message. A holdout compares message against no message. Some of the people who buy after an email would have bought anyway; the holdout shows how many.
How it works: randomly hold back, say, 10% of a flow or campaign audience, then compare their purchase rate with everyone who got the message. The difference is the incremental lift.
In SmartyTap, A/B tests compare versions and don't create a no-message group automatically. You can approximate one by excluding a segment you set aside from a campaign and comparing its orders with the people who got the message.
Example
A win-back flow goes to 90% of lapsed customers; 10% get nothing. After 30 days, 6% of the messaged group and 4% of the holdout reordered: a 2-point lift.
Why Holdout group Matters
Attribution counts every order after a click; a holdout tells you which orders the message actually caused.
Holdout group FAQ
How big should a holdout group be?
Often 5 to 20% of the audience, large enough to show a difference in orders.
Isn't holding people back lost revenue?
A little, for a while. It's the price of knowing whether a message earns more than it costs.
Explore More Terms
Browse our complete glossary of e-commerce and subscription terms.
