Write a hypothesis that states the change, the effect you expect, the measure and the reason.
Before you change anything else on your website, write down the last change you made, what happened in the week after it, and anything else that was different that week. If that list has more than one item, you cannot say what caused the result. A fair test is set up so that you can.
Mei Ling bakes custom birthday cakes from her flat in Punggol. Last month she changed her order page in four ways: a new photo at the top, a shorter form, a different button colour, and a line saying orders need five days' notice. The following week, orders went up. She was pleased but puzzled. She did not know which of the four changes caused it, or whether the changes caused it at all, because that was the week before a long weekend, when cake orders usually rise anyway.
Making a change and then watching what happens is not a test, because you cannot tell what caused the result. A test is set up so that when the result arrives, you know what caused it, and that setup starts before anything is changed.
The usual method is an A/B test: you show two versions, A and B, to similar groups of people at the same time and compare how each group behaves. Both groups see their version in the same week under the same conditions, so a difference in results comes from the difference between the versions or from chance. Lesson 7.2 deals with chance as an explanation.
Before changing anything, write a hypothesis, the sentence that says what you expect and why. It has three parts. The "if" names one specific change, so you know exactly what version B is. The "then" names the measure and the direction you expect it to move, so you know what to look at when the test ends. The "because" names the reason.
People often leave out the "because", but it teaches the most. If the test wins, your reason is probably right and you can apply it elsewhere. If it loses, the reason was wrong and you have learned something about your customers. A vague plan such as "let's try a new product page and see" gives you a result only, while a hypothesis gives you a result plus an explanation you can check.
Farah runs an online workwear shop. In lesson 4.3 her funnel showed most mobile visitors leaving the trousers page without adding anything to the cart. She found the size chart was hidden in a pop-up that was hard to close on a phone. Her hypothesis: if we show the size chart directly on the product page instead of in a pop-up, then the share of product page visitors who add to cart will rise, because shoppers on mobile cannot easily check their size today.
Each test changes one thing. Mei Ling's page differed from the old one in four ways, so even in a proper A/B test, a win for B would not tell her which change helped. One may have helped a lot while another hurt a little, and her next redesign would have nothing to go on.
One thing can be large. A whole new layout for an order form counts, as long as it is one idea you can name. Avoid bundling unrelated changes, such as a new photo, a new price and a new button, into one version. The rule holds on your website, in ads and in email. The course "Paid ads: Meta, Google and TikTok" makes the same point in lesson 9.1, "Test one thing at a time and decide in advance", and the course "Email marketing and automation" makes it in lesson 8.3, "Test one change at a time".
Alongside the hypothesis, write down two more things before the test starts. The first is the main measure, the single number that decides the test. Choose one close to the change. A subject line affects opens and clicks more directly than sales, and a product page change affects add-to-cart more directly than monthly revenue. Farah's main measure is the add-to-cart rate on the product page. For Mei Ling's order form it would be form submissions per visitor to the page. You can watch other numbers, but only the main measure decides the outcome.
The second is the stopping rule, which says when the test ends: after a set number of visitors or recipients per version, or after a set number of full weeks, whichever you worked out in advance. Lesson 7.2 shows how to estimate that number.
With both written down, you will not check results each morning, see B ahead on some number and declare victory early. You show A and B to similar groups at the same time, end the test when you said you would, and judge it by the measure you chose. Then go back to your "because" and note what the result says about your reason.
The size of the change decides how long the test takes. Small tweaks, such as a slightly different button shade or one word in a headline, usually make small differences, and those are very hard to tell apart from chance without a very large number of visitors. Lesson 7.2 shows how quickly the numbers you need grow. Bold changes tend to make bigger differences, and bigger differences show up sooner.
For a small business, test things customers would notice: a different offer, a new page structure, a shorter form, free delivery above a threshold, or a different first photo. Run each idea through three checks: it changes one named thing, a customer would notice it, and you have a reason to expect it to work. Mei Ling's long weekend notice line is a decent test candidate, and her button colour is not.
Look at your own website, emails and ads with those checks in mind, and the ideas worth testing tend to stand out from the ones that are just tinkering.
Write three hypotheses for your website, emails or ads using if, then, because. Name the main measure for each one.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).