You will plan, run and write up one A/B test on a page, email or ad with a decision at the end.
Most small businesses that try A/B testing run one test, watch it for a few days, pick whichever version is ahead and never write anything down. Six months later, nobody remembers what was tested or why the page looks the way it does. Someone suggests the same idea again, and the cycle repeats.
This exercise runs one test properly from start to finish and leaves a one-page record behind. The worked example is Mei Ling's, for her home bakery in Punggol. She tested in email, because lesson 7.3 showed that was the one place her traffic could finish a test within weeks.
Start from one of the hypotheses you wrote in lesson 7.1, A test starts with a hypothesis, not a hunch. Choose one that is bold enough to detect with your traffic, using the calculator from lesson 7.2.
Mei Ling's hypothesis: if the monthly newsletter leads with an offer of free delivery within Punggol for orders placed two weeks ahead, instead of leading with photos of last month's cakes, then the share of recipients who click through to the order page will rise, because parents who plan ahead are put off by the delivery charge at the end of the form.
Main measure: click-through rate to the order page. Her current rate is about 4 percent (an example figure). She wanted to detect a rise to 8 percent, and the calculator gave about 553 recipients per version. Her list of about 2,400 subscribers, split in two, gives about 1,200 per version, comfortably above that.
Stopping rule: the result is read seven days after the send, once nearly all clicks have come in, and not before.
Use the testing feature built into the tool where the test runs. Email tools usually let you create two versions of a campaign and split the list at random. Ad platforms have their own experiment or A/B test features. For a website, some website builders and shop platforms include page testing, and stand-alone testing tools exist for sites that do not.
Whatever the tool, check four things before launch: that the split is random rather than by sign-up date or alphabet, that each person sees only one version, that both versions go out at the same time, and that the change in your hypothesis is the only difference between them.
Mei Ling built two versions of the newsletter that were identical apart from the top section. She switched off the email tool's option to send the winner automatically to the rest of the list, because she was sending to everyone at once, and she wanted to judge the result herself against her own rule.
Once the test is live, leave it alone. Do not edit either version midway, and do not stop early because one version looks ahead on day two. Lesson 7.2 explained why early leads fade.
When the stopping rule is reached, record the numbers for each version: how many saw it, how many did the main measure, and the rate. Then put the numbers into a free A/B test results calculator to see whether the gap is bigger than chance would usually produce.
Mei Ling's result, with example figures: version A, the photos, went to 1,200 people and got 48 clicks to the order page, a rate of 4.0 percent. Version B, the delivery offer, went to 1,200 and got 62 clicks, 5.2 percent. B was ahead. But the calculator said a gap that size would turn up by chance about 17 percent of the time even if the versions were equally good, well short of the 95 percent confidence level she had chosen, so the test found no clear winner.
That is a real result, and it gets recorded like any other. It says that if the offer helps, it helps by less than she set out to detect.
A test ends with one of three decisions. Keep the new version, if it won clearly. Roll back to the original, if it lost or if the change has costs. Or test again, with a bolder change or a bigger sample, if the result was unclear and the idea still seems worth pursuing.
Mei Ling chose to test again. Free delivery costs her money on every order, so she did not want to roll it out without clear evidence. Next month she will test a bolder version: the same offer in the subject line as well as the top of the email, with orders placed rather than clicks as a second measure she will watch.
Her write-up fits on one page, under six headings: hypothesis, setup (tool, split, dates, what differed), planned sample size and stopping rule, result with the numbers for each version, decision, and what she will test next. She keeps it in a folder called "tests", one page per test, so the next idea can be checked against what has already been tried.
A finished test has a written hypothesis with a reason, a planned sample size that was reached, a result recorded with real numbers even when nothing won, a decision someone acted on, and a next step. Whether it produced a winner is beside the point, since a test that rules out an idea you were about to spend money on has paid for itself.
Your own test may run in email, in an ad platform or on a page. Whichever you choose, the one-page write-up at the end is what turns a single experiment into something the business remembers.
Produce a one-page test write-up with the hypothesis, setup, sample size, result, decision and what you will test next.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).