A/B tests: SmartyTap docs

Every campaign is a chance to learn what your customers answer. An A/B test sends different versions of a message to a small test group, waits, and sends the version that did best to everyone else. The next campaign starts from what worked, not from a guess.

Test an email

Open the setup

Select the email in your campaign, open Campaign setup and, under A/B test, click Create A/B test.

Add versions

Click Add version, give it a Version name, and use Editing to switch between versions. Each version is a full email, so you can change the subject line, preview text or content. You can have up to seven.

Set the test

Choose the Test size, what to Pick the winner by, and the Test length (hours), as described below.

Schedule

Review and schedule the campaign as usual. The message shows A/B test running while the test runs.

Test a text

In the SMS composer, click Create A/B test, then Duplicate variation and change the wording. The test settings are the same as for email.

Test settings

Setting	What it does	Starts at
Test size	The share of the audience that gets a version during the test, from 10% to 100%. The rest get the winner.	20%
Pick the winner by	Open rate, Click rate or Placed order rate for email. Texts use Placed order rate.	You choose
Test length (hours)	How long to wait before picking, from 1 to 72 hours.	6 hours for opens and clicks, 24 for orders

Which winner to pick by

Testing subject lines? Use open rate. Testing the content or offer? Use click rate, or placed order rate if you have enough orders to tell versions apart. Placed order rate counts people who ordered after getting a version; it isn't revenue. Click rate needs Track link clicks turned on.

How the winner is sent

When the test length is up, the best version goes to everyone who wasn't in the test. People waiting show as Waiting for test results.
If the results tie, Version A goes to everyone else.
To decide early, open Campaign performance, find A/B test results and click Send this version to everyone else on the version you want. Choose Keep testing to change your mind.
With a test size of 100%, everyone gets one of the versions and no winner is sent. Use it to compare versions across the whole audience.

Tests and local time

A test that picks a winner sends at one time, so Each recipient's timezone is turned off for it. Choose a specific time zone, or set the test size to 100% to split versions and still send in each recipient's local time.

FAQ

Can I test several things at once?

Each version can differ in as many ways as you like, but SmartyTap compares whole versions. It doesn't run multivariate tests that measure each change separately, so change one thing at a time if you want to know what made the difference.

How big should the test be?

Big enough that each version reaches a few hundred people. With a small list, a larger test size or a 100% split teaches you more.

Will anyone get two versions?

No. People in the test group get one version, and only people outside the test get the winner. Messages from the same campaign also don't count toward Smart Sending.

Guides

On the product page

Campaigns with an automatic A/B winner

A/B tests