Plan, run and write up one A/B test on a page, email or ad, ending with a decision.
Open the hypotheses you wrote for lesson 7.1, "A test starts with a hypothesis, not a hunch", and pick the one you most want answered. You'll run one complete test on it and finish with a one-page record of what happened.
Small businesses often run one test, watch it for a few days, pick whichever version is ahead and write nothing down. Six months later nobody remembers what was tested or why the page looks the way it does. Then someone suggests the same idea again and the cycle repeats.
Your hypothesis needs three parts: the change ("if..."), the effect you expect on a measure ("then..."), and a reason ("because..."). The change also has to be bold enough for your traffic to detect. Check this with the sample size calculator from lesson 7.2.
Set one main measure, the single figure the test is judged on. Note its current rate, the target rate you want to be able to detect, and the sample size per version the calculator gives. Then confirm your audience is big enough to supply that many people for each version. Last, set a stopping rule: a fixed point at which you read the result, and not before.
Mei Ling runs a home bakery in Punggol. She tests in email because lesson 7.3 showed it was the one place her traffic could finish a test within weeks. Her hypothesis: if the monthly newsletter leads with free delivery within Punggol for orders placed two weeks ahead, instead of leading with photos of last month's cakes, then click-through to the order page will rise, because parents who plan ahead are put off by the delivery charge at the end of the order form.
Her main measure is click-through rate to the order page. It is currently about 4 percent (an example figure, as are all of hers), and she wants to be able to detect 8 percent. For that, the calculator gives about 553 recipients per version. Her list of about 2,400 subscribers splits into about 1,200 per version, comfortably above 553. Her stopping rule is seven days after the send, when nearly all clicks are in.
Use the testing feature built into the tool where the test runs. Email tools usually let you create two versions of a campaign and split the list at random. Ad platforms have their own experiment or A/B test features. Some website builders and shop platforms include page testing, and there are stand-alone testing tools for sites without it.
Whatever the tool, check four things before launch. The split is random, not by sign-up date or alphabet. Each person sees only one version. Both versions go out at the same time. The change in your hypothesis is the only difference between them.
Mei Ling builds two newsletters that are identical apart from the top section. Her email tool has an option to auto-send the winner to the rest of the list. She switches it off, because she is sending to everyone at once and wants to judge the result herself against her own rule. If you want to make that call yourself, turn it off too.
Once the test is live, leave it alone. Don't edit either version midway, and don't stop early because one version looks ahead on day two. Lesson 7.2 explained why early leads fade.
At the stopping rule, record three things for each version: how many saw it, how many did the main measure, and the rate. Put those numbers into a free A/B test results calculator to see whether the gap is bigger than chance would usually produce.
Version A of Mei Ling's newsletter (photos) went to 1,200 recipients and got 48 clicks, 4.0 percent. Version B (delivery offer) went to 1,200 and got 62 clicks, 5.2 percent. B is ahead. The calculator says a gap that size would appear by chance about 17 percent of the time if the two versions were equally good, which falls well short of the 95 percent confidence level she had chosen.
The result is no clear winner: at the chosen confidence level, the gap is not bigger than chance would usually produce. Record it the same way you would record a win. It means that if the change helps, it helps by less than the test was set up to detect.
A test ends with one of three decisions. Keep the new version if it won clearly. Roll back to the original if it lost or the change has costs. Test again with a bolder change or a bigger sample if the result is unclear and the idea still seems worth pursuing. Don't roll out a change that has an ongoing cost without clear evidence.
Mei Ling decides to test again. Free delivery costs her money on every order, so she won't roll it out without clear evidence. Next month's test is bolder. It puts the same offer in the subject line as well as at the top of the email, and she will also watch orders placed, not just clicks, as a second measure.
The write-up fits on one page under six headings:
Hypothesis Setup (tool, split, dates, what differed) Planned sample size and stopping rule Result, with the numbers for each version Decision What to test next
Keep all your write-ups in one folder, one page per test. Then you can check any new idea against what has already been tried. Mei Ling writes hers under the six headings and files it in a folder called "tests".
A finished test has five things: a written hypothesis with a reason, a planned sample size that was reached, a result recorded with real numbers even when nothing won, a decision someone acted on, and a next step. Whether the test produced a winner doesn't matter. A test that rules out an idea you were about to spend money on has paid for itself.
Now take your hypothesis through all four steps. Choose it and its sample size, set it up in the right tool, let it run to the stopping rule and record the result, then make your decision and write the page. Your own test may run in email, in an ad platform or on a page. Whichever you choose, the one-page write-up at the end is what turns a single experiment into something the business remembers.
Write a one-page report on your test covering the hypothesis, setup, sample size, result, decision and what you will test next.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).