Why experiments beat observation

You will be able to explain why randomised experiments give strong evidence of cause, and when you can't run one.

Take a sheet of paper and note one practice at your workplace that everyone says works. It could be a sales technique, a training course or a new process. Underneath it, write down how anyone knows that it works.

Most often the answer is that people who used it got better results. Evidence of that sort can't tell you whether the practice produced those results, because the people who chose it may have been different from the start. To find out whether something caused a result, the strongest tool you have in everyday life is the randomised experiment, and you can run a simple version of one at work.

Letting chance pick the groups

A randomised experiment is one where you decide who receives the treatment, and you make that decision by chance. The alternative is to study people who opted in themselves. When you only watch what people chose to do, you are collecting observational data, and the groups in observational data are full of hidden differences. You can adjust for the differences you know about, but you can't be certain you've caught every one.

Lesson 4.1, "Three other reasons two things move together", worked through this with a retailer's app. App users might be more loyal, richer or younger than other customers, or they might be furnishing a new flat. Any of those could account for their higher spending.

With random assignment, you start with a single group and divide it by chance into a treatment group and a no-treatment group. A coin flip works, and so does a random number in a spreadsheet. Because chance sends each person to one side or the other, confounders end up spread roughly evenly across the two groups. That includes the confounders you know about and the ones nobody has thought of. The even spread only holds if the groups are large enough. When they are, the treatment is the only systematic difference left between the groups. If the results differ by more than chance can account for, the treatment is the most likely cause.

Medical research depends on this design. Before a new treatment is approved, it goes through randomised controlled trials. Inside a product, the same design is called an A/B test. A website or app shows one version to a random half of its visitors and another version to the other half, then compares what the two halves do. You have almost certainly been part of many A/B tests without knowing it. In every setting, a coin decides who gets what, and nobody picks.

How Daniel tested the app

Daniel works for a retailer that sells through a website and an app, and wants to find out whether the app makes people spend more. Daniel's manager isn't persuaded by the questions raised in Lesson 4.1. She points out that they could argue about confounders forever, and asks how they would actually know whether the app makes people spend more. This is a fair challenge, and the clear answer is to run a randomised test. Daniel's version went like this (all figures here are invented for illustration):

Choose one pool of people. Daniel takes 8,000 recent website customers who don't have the app. Divide them by chance, with a coin flip or a spreadsheet random number. Daniel ends up with two groups of 4,000. Give the treatment to one group, and give the other group the same thing without the treatment. Group B receives an email with a voucher for downloading the app. Group A receives the same email with no mention of the app. Wait for a fixed period. Daniel waits three months. Compare one chosen outcome across the groups. Daniel counts the customers who placed a large order. Check whether the gap is bigger than chance would explain, using an analytics tool or an online calculator.

After three months, 120 customers in Group A have placed a large order, which is 3 percent of 4,000. In Group B the figure is 150, or 3.75 percent. The gap is 0.75 points, and 0.75 points divided by 3 percent is a 25 percent relative rise.

Customer loyalty can't account for this difference, and neither can people buying for new flats. The random split put loyal customers and new-flat buyers into both groups in roughly equal numbers.

Ruling out luck

Random assignment removes the worry that the groups started out different. It does not rule out the possibility that the difference is down to chance. In absolute terms, Daniel's gap is 30 extra customers (150 minus 120), and a number that small could still be luck. A proper test works out how likely a gap of a given size would be by luck alone. Analytics tools and online calculators can do this check, and anyone who runs A/B tests regularly should learn how to use them.

You can use the same design for ordinary questions at work. If you want to know whether a new onboarding email cuts support calls, send it to a random half of new customers and compare the support calls from each half. If you want to know whether a different subject line gets more replies, split your mailing list at random and send one version to each half.

Evidence when an experiment is impossible

Many causal questions can't be tested by experiment. Nobody can randomly assign people to smoke, to grow up in a particular neighbourhood, or to live through a recession. In those cases, two kinds of evidence come closest: natural experiments, and many studies that point the same way.

In a natural experiment, something outside anyone's control has divided people into groups in a way that is close to random. One example is a policy that applies only to people born after a certain date. Another is a change that rolls out in one area before another for administrative reasons. Either can produce groups that differ mainly in the one thing you care about. Researchers search for natural experiments carefully, and when they find a good one, the evidence can be nearly as strong as an experiment.

The second kind of evidence builds up when different teams find the same link using different data and different methods in different countries. It then becomes harder to explain the link away with any single confounder. Every study has weak points, but it is rare for all of them to share the same one.

Mechanisms as a reason to test

A mechanism is an account of how a cause would produce its effect. A plausible mechanism makes a claim more believable and shows that it isn't absurd. It also gives you something specific to check. In Daniel's case, one possible mechanism is the app's push notifications, which remind customers to come back and buy. Daniel could test this by checking whether app users who turned notifications off spend less.

A convincing explanation of how something might work is not evidence that it does work. Plenty of treatments with convincing mechanisms have failed in trials. Use a mechanism as a reason to take a claim seriously enough to test it.

Go back to the claim you noted at the start. Most workplaces have one that everyone repeats and nobody has tested. In the activity below, design the comparison that could settle yours: the pool of people, how you'll split them by chance, what each group gets, how long you'll wait, and which outcome you'll compare.

Pick one causal claim at your workplace and describe how you would test it with a simple A/B comparison, including how you would assign the groups.

Course

Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).