You will be able to write a hypothesis that names the change, the expected effect, the measure and the reason.
Mei Ling bakes custom birthday cakes from her flat in Punggol. Last month she changed her order page: a new photo at the top, a shorter form, a different button colour and a line saying orders need five days' notice. The following week, orders went up. She was pleased and slightly puzzled, because she had no idea which of the four changes did it, or whether it was the changes at all. It was also the week before a long weekend, when cake orders usually rise anyway.
She had made a change and then watched what happened. A test is set up so that when the result comes in, you know what caused it. That starts before anything is changed, with a hypothesis.
An A/B test shows two versions of something, A and B, to similar groups of people at the same time, and compares how each group behaves. Because both groups see their version in the same week, under the same conditions, a difference in results can be put down to the difference between the versions, or to chance, which lesson 7.2 deals with.
A hypothesis is the sentence that says what you expect and why. A useful format has three parts:
If we change this, then this measure will move, because of this reason.
Each part does a job. The "if" names one specific change, so you know exactly what B is. The "then" names the measure and the direction you expect, so you know what to look at when the test ends. The "because" names the reason. It is often left out, yet it teaches you the most. If the test wins, the reason is probably right and you can apply it elsewhere. If it loses, the reason was wrong, and you have learned something about your customers.
Here is Farah's, from her online workwear shop. In lesson 4.3 her funnel showed most mobile visitors leaving the trousers page without adding anything to the cart, and she found the size chart hidden in a pop-up that was hard to close on a phone. Her hypothesis: if we show the size chart directly on the product page instead of in a pop-up, then the share of product page visitors who add to cart will rise, because shoppers on mobile cannot easily check their size today.
Compare that with "let's try a new product page and see". The vague version gives you a result. The hypothesis gives you a result and an explanation you can check.
If B differs from A in four ways, as Mei Ling's page did, and B wins, you do not know which change helped. One may have helped a lot while another hurt a little. Next time you redesign, you have nothing to go on.
So each test changes one thing. "One thing" can be a large thing, such as a whole new layout for the order form, as long as it is one idea you can name. What you avoid is bundling unrelated changes, such as a new photo, a new price and a new button, into one version.
Paid ads: Meta, Google and TikTok makes the same point for ads in lesson 9.1, Test one thing at a time and decide in advance, and Email marketing and automation does it for emails in lesson 8.3, Test one change at a time. The rule holds for any channel you test in.
Before the test starts, write down two more things.
The main measure is the single number that decides the test. For Farah's size chart, it is the add-to-cart rate on the product page. For Mei Ling's order form, it would be form submissions per visitor to the page. Choose a measure close to the change: a subject line affects opens and clicks more directly than sales, and a product page change affects add-to-cart more directly than monthly revenue. You can watch other numbers too, but only the main measure decides.
The stopping rule says when the test ends: after a set number of visitors or recipients per version, or after a set number of full weeks, whichever you have worked out in advance. Lesson 7.2 shows how to estimate that number.
Writing these down first protects you from yourself. Without them, it is very tempting to look at the results each morning, notice that B is ahead on some number, and declare victory. With them, the test ends when you said it would, judged by the measure you chose.
The size of the change you test decides how long the test takes. A small tweak, such as a slightly different shade of button or one word in a headline, usually makes a small difference, and small differences are very hard to tell apart from chance without a very large number of visitors. Lesson 7.2 shows how quickly the numbers grow.
Bold changes tend to make bigger differences, and bigger differences show up sooner. For a small business, that means testing things customers would notice: a different offer, a new page structure, a shorter form, free delivery above a threshold, a different first photo. Mei Ling's long weekend notice is a decent candidate. Her button colour is not.
A good test idea passes three checks. It changes one named thing, a customer would notice it, and you have a reason to expect it to work. Look at your own website, emails and ads with those checks in mind, and the ideas worth testing tend to stand out from the ones that are just tinkering.
Write three hypotheses for your website, email or ads in the if, then, because format, each with its main measure.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).