Explain why small tests give misleading results, and use a sample size calculator before you start a test.
Before you start any A/B test, put your numbers into a free A/B test sample size calculator and find out how many visitors the test needs. If you do this first, you won't act on a result that chance produced.
Farah started the size chart test from lesson 7.1 on her trousers page on a Monday. By Wednesday, version B had 9 add-to-carts from 41 visitors and version A had 4 from 39 (invented example figures). B looked more than twice as good, and she was ready to switch every page to B that afternoon. By the end of the second week, the two versions were almost level.
Nothing was wrong with the test, and the early lead was down to chance. Early results swing about by chance, and an early winner often fades as more visitors arrive. You can think of each visitor as a coin toss that comes up "adds to cart" some of the time. Even when two versions are equally good, a few dozen visitors will rarely split evenly between them, and the version that gets a lucky run looks like the winner.
These swings are bigger than most people expect. Suppose the true add-to-cart rate is 12 percent (an invented example figure). With 40 visitors, the measured share will usually land somewhere between about 2.5 percent and 22.5 percent by chance alone, which is roughly 1 to 9 people. Farah's 9 out of 41 against 4 out of 39 is the kind of gap that chance alone produces. As visitor numbers grow, the swings shrink and the measured rate settles closer to the true rate.
Sample size decides what a test can tell you. A large real difference between versions shows through the chance swings quickly, while a small one stays buried in noise until there are many visitors.
Farah's numbers show how quickly the required sample grows. Her current add-to-cart rate is 12 percent, and the trousers page gets about 450 visitors a week, split between the two versions (both invented example figures). Her sample sizes come from a standard sample size formula for comparing two rates, using a 95 percent confidence level and 80 percent power, the standard settings most calculators start with:
To detect a rise from 12 to 18 percent: about 555 visitors per version, roughly 1,110 in total, about two and a half weeks of traffic. From 12 to 15 percent: about 2,036 per version, roughly 4,070 in total, about nine weeks. From 12 to 13 percent: about 17,169 per version, over 34,000 in total, about 76 weeks, which is a year and a half for one test.
As a rule of thumb, halving the size of the difference you want to detect roughly quadruples the sample you need. A small business can't run tests that look for tiny differences, so lesson 7.1 tells you to test changes big enough to matter.
When a testing tool calls a result statistically significant, it is answering a narrow question: if there were really no difference between A and B, how unlikely would a gap this big be from chance alone? When the answer is "very unlikely", the tool declares a winner.
Most tools use a 95 percent confidence level. At that level you accept a 5 percent chance of naming a winner when the versions are really the same. Power works from the other direction. It is usually set at 80 percent and is the chance that the test spots a real difference of the size you chose, if one exists. Both settings are habits of the trade and can be changed, though stricter settings need a bigger sample.
A significant verdict doesn't tell you how big the effect is, whether it matters to the business, or whether the winner will keep winning. A confirmed result on a tiny difference may not be worth acting on. A test that names no winner doesn't prove the versions are equal either. It means the test could not tell them apart with the visitors it had.
The sample size calculation assumes you look at the result once, at the end. Suppose you check every day and stop as soon as one version looks clearly ahead. Each check gives chance another opportunity to produce a lucky streak, and one streak will often cross the line. The more often you peek and are willing to stop, the more likely it is that your "winner" is a fluke.
Lesson 7.1 asks you to set a stopping rule before you start. Farah's was to run until each version had at least 555 visitors and the test had covered three full weeks, whichever came later. She could look at the numbers along the way, but she could not stop early because she liked what she saw.
You don't need to do the maths yourself. Free A/B test sample size calculators are easy to find online, and many testing tools include one. Start by finding your current conversion rate for the measure you are testing. GA4 can give you this, along with the weekly traffic for the page. In GA4, add the Key events metric to the report for your page, then select the event you are testing from the drop-down menu so the report counts only that one key event. Divide that count by the page's visitors over the same period to get your conversion rate. Next, decide the smallest change you want to be able to detect, and choose one big enough to matter. Enter both into the calculator with the confidence and power settings, which you can usually leave at 95 and 80 percent.
The calculator gives you the number of visitors, or recipients, needed per version. Work out the total across both versions, divide it by your weekly traffic to the page, and round up to full weeks to get how long the test must run. Before you begin, set your stopping rule as a minimum number of visitors per version and a minimum number of full weeks, whichever comes later. You can watch the results while the test runs, but don't stop early because one version looks ahead.
A very long answer, such as six months, is still useful. It tells you to test something bolder, or to test somewhere with more traffic, which lesson 7.3 covers.
Your own homepage is the obvious first page to try this on. Get its key events for the event you care about and its weekly traffic from GA4, work out its current conversion rate, choose the smallest change worth detecting, and put the numbers into a calculator to see how many weeks the test would need.
Enter your own traffic and conversion rate into a free sample size calculator. Write down how many weeks a homepage test would take.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).