Email A/B Testing: What to Test, How Long to Run It and How to Pick a Winner

To A/B test an email, send two or more versions to random groups from the same audience, change one thing at a time, wait a few hours for clicks or about a day for orders, then send the winner to everyone else. Start with the subject line and the offer, and judge by clicks or orders where you can.

Every campaign you send is a chance to learn what your customers respond to. An A/B test turns that chance into an answer: send two or more versions of a message to a small part of your list, wait, and send the version that did best to everyone else. Do it every week and your emails get better from evidence, not from guesses.

This guide covers what to test first, how big the test should be, how long to wait, which number should pick the winner, and the mistakes that make results meaningless.

An A/B test (or split test) sends different versions of one message to random groups from the same audience and compares a single result, such as clicks. Because the groups are random and receive the message at the same time, the difference in results comes from the message, not the audience. With a "test and send" setup, the test goes to a share of the list, and the winner goes to the rest.

Start with the things that change the most people's behavior, and test one thing at a time so you know what made the difference.

• Open rate: Only for subject lines, preview text and sender names, and read with care. Apple's Mail Privacy Protection downloads email content in the background by default (checked October 19, 2026), so many Apple Mail opens are recorded whether or not anyone read the email.
• Click rate: The most reliable signal for content, images and buttons. It needs link tracking on.
• Placed order rate: Best for offers and anything that should change buying, but it needs enough orders to tell versions apart. It counts people who ordered, not how much they spent.

The fewer people who take the action you measure, the bigger the test must be. If 2 people in 100 click, a test group of 200 per version gives you about 4 clicks each, and the "winner" is luck. As a rough rule, aim for at least a few hundred people per version for opens and clicks, and far more when you pick by orders.

• Large list: Test on 10% to 20% of the audience and send the winner to the rest.
• Medium list: Test on a larger share, such as 30% to 50%, so each version gets enough people.
• Small list: Split the whole list between versions and treat the result as a lesson for next time, not a winner for this send. Repeat the same test over several campaigns before deciding.

Most clicks happen in the first hours after a send, and orders take longer. A few hours is usually enough when you pick by opens or clicks; give orders about a day. Don't wait so long that the winner arrives after the moment has passed: a test that ends at 11 p.m. sends the winner to everyone at 11 p.m.

• A small gap on a small test is noise. If version B got 31 clicks and version A got 28, you learned nothing yet.
• Look for differences that repeat across campaigns before changing your whole approach.
• Check the downside too: a winning subject line that also doubles unsubscribes isn't a winner.
• Write down every test and its result. A simple shared sheet becomes your playbook.

• Changing several things at once: If the subject, image and offer all differ, you know which email won but not why. Testing many combinations at once (multivariate testing) needs a much bigger list.
• Picking by opens for an offer test: The subject line drives opens; the offer drives orders. Measure what you changed.
• Testing on a biased slice: Every version must go to a random part of the same audience at the same time.
• Stopping early: An early lead often fades. Decide the test length before you start.
• Never testing again: Customers change. A subject line style that won last spring may not win now.

Texts can be tested too, with two differences. Phones don't report when a text is read, so there are no opens: pick the winner by clicks or orders. And texts are billed per part, so keep each version the same length, or you are also testing cost.

• Offer first vs product first.
• With the customer's first name vs without.
• A question ("Still thinking about the boots?") vs a statement.
• Short link at the end vs in the middle.

Automations deserve tests too, because they run every day. Split new subscribers at random between two welcome offers, or two waits before an abandoned checkout reminder, and compare orders over a few weeks.

SmartyTap campaigns include A/B tests with an automatic winner, for email and texts.

• Up to seven versions: Each version is a full email, so you can change the subject line, preview text or content. For texts, duplicate a variation and change the wording.
• Your choice of winner metric: Open rate, click rate or placed order rate for email; texts use placed order rate.
• Test size and length: Test on 10% to 100% of the audience, for 1 to 72 hours. At 100%, everyone gets a version and no winner is sent.
• Automatic or manual winner: The best version goes to everyone else when the test ends, or you can send a version to everyone early. If results tie, Version A is sent.
• Random splits in flows: A Conditional split can send a random sample down each path to test two flow branches against each other.

SmartyTap compares whole versions; it doesn't run multivariate tests. Step-by-step setup is in A/B tests.

Glossary: A/B test, holdout group, click-through rate and SMS campaign. Read next: email marketing benchmarks explained, email segmentation examples and the Black Friday email and SMS checklist.

A few hours is usually enough when you pick the winner by opens or clicks, because most activity comes soon after sending. When you pick by orders, give it about a day. Decide the length before the test starts.

Start with the subject line and the offer, because they change the most behavior. Then test the main image, the call to action and the length. Change one thing at a time.

Only for subject line tests, and with care. Apple Mail Privacy Protection downloads email content in the background, so opens are inflated. Click rate and placed order rate are more reliable.

An A/B test compares whole versions of a message. A multivariate test measures several changes at once and how they combine, which needs a much larger audience to give a clear answer.