Set up a fair A/B test on an email and tell when a result is too small to trust.
Priya sends a newsletter to 300 parents (an example list). In one issue she changed three things at once: she used a question as the subject line instead of a statement, sent on Tuesday instead of Thursday, and used a button instead of a text link.
That issue got more clicks than any before it. She was delighted, then stuck. She could not tell which of the three changes had worked, or whether to keep all of them. What she had sent was a different email. Changing several things at once does not give you a test.
Testing shows you what works for your own list. That can be different from what someone online says works. A careless test gives you confident answers that are wrong, so the method needs care.
An A/B test, also called a split test, sends two versions of an email to two randomly chosen parts of your list. The versions differ in one way, and you compare the results. Most email tools have a built-in feature for this, usually labelled A/B test or split test, and it does the random split for you.
The two versions must differ in only one way. Say version B has a different subject line and a different send time, and B wins. You cannot tell which change caused it. Good single changes to test include:
the subject line the preview text the send day or time the wording of the call to action the position of the main link plain text versus a designed template
Pick the one change you most want to learn about and keep everything else identical.
Before you send, write down which number decides the winner and what result would make you act. Deciding in advance stops you picking whichever number happens to favour the version you liked.
For most email tests the measure should be clicks. Opens are unreliable because privacy features count many emails as opened that nobody read, as lesson 8.1 explained. A subject line test judged on opens may just reward whichever version went to more Apple Mail users. A click only happens when a person acts. If your email asks for replies, replies are a valid measure too. Bookings or orders traced through UTM tags, from lesson 8.2, are a better measure still. On a small list, though, they are usually too few to decide a test on their own.
On a small list, most differences between two versions are chance. You could send the same email twice to two random halves of your list and still see a gap of a few clicks. The gap would come from things like which readers happened to be free that evening.
Say Priya runs a proper test. She splits her 300 parents into two halves of 150. Version A gets 12 clicks from 150 people, which is 8 percent. Version B gets 15 clicks from 150 people, which is 10 percent. B looks better, but the difference is 3 clicks. A gap that tiny could easily appear between two identical emails by chance.
The working rule is that a difference of a few clicks is usually chance. The smaller the list, the bigger the gap has to be before it means anything. Some email tools show a confidence figure beside a test result. Use it if yours does, and be cautious if it does not. Module 7 of the course "Marketing analytics: GA4, attribution and testing" covers sample size properly. Lesson 7.3 of that course, "Testing when you do not have much traffic", deals directly with small lists.
For email on a small list, the habit of repeating winners helps more than any formula. Do not turn a single win into a rule. If one version wins, try the same kind of change on your next two or three sends. If it wins three times out of three, you have learned something about your readers. If it wins once and loses twice, the first result was probably chance.
Record every test in a test log kept alongside your send log from lesson 5.5. Give it columns for the date, what changed, version A, version B, the measure, the result for each, and what you concluded. Over a few months it builds into a record of what works for your list in particular. That record is more useful than general advice.
On a very small list, each half of a split may be too small for one send to teach you anything. In that case, skip the split and compare one issue against another over time. Alternate the two approaches across several months and read the pattern in the log. This is slower than a split test but far more honest about how your readers respond.
Some changes need no test because the answer is already clear. Fix a broken link, missing alt text or a fake "RE:" in a subject line straight away.
In the activity below you will write a test plan for your next send: the one change, the measure, how you will split the list, and what result would change what you do in your following email.
Write a test plan for your next send. Name the one change, the measure, how you will split the list and what result would change your next email.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).