Judge a causal claim

You will be able to assess a real causal claim from the news, a report or your workplace.

Someone is always telling you that one thing produces another: that a supplement lowers blood pressure, that a new rota raises output, that a course gets people promoted. These claims usually come with a confident figure, and few readers ask whether the figure shows cause at all. If you run the check on paper, in the same order each time, you end up with a verdict you can defend. You also have working that anyone who disagrees can look over. The check uses everything covered in the module.

What to have ready and where claims turn up

Allow about thirty minutes. You need one causal claim, paper or a spreadsheet, and the four alternatives from Lesson 4.1, "Three other reasons two things move together". Those four are confounders, reverse causation, selection and chance.

You'll find causal claims in health articles, management books, company reports, internal newsletters and policy announcements. The wording gives them away. Words such as "boosts", "leads to", "improves", "linked to" and "the secret of" either say outright that one thing causes another or hint at it. Be extra suspicious of "linked to". Writers often choose it because it suggests cause while keeping them technically safe.

Rewriting the claim as one narrow sentence

Before you can judge a claim, rewrite it in the form "X causes Y" and make it as narrow as it will go. "Mentoring helps careers" is too loose to judge. It never says what "helping" means, and it gives you nothing to compare against. A tighter version is: joining the company mentoring programme causes employees to be promoted sooner than they otherwise would have been. You can hold that version up against evidence.

The five rows of the worksheet

Draw up five rows and label them, from top to bottom: claim, evidence, alternative explanations, what would settle it, verdict.

The narrow "X causes Y" sentence goes in the first row.

The second row records what the claim rests on and what kind of evidence it is. Observational evidence comes from watching what people chose to do. Experimental evidence comes from a study where someone else decided who got the treatment, ideally by chance. Experimental evidence counts for more, and Lesson 4.2, "Why experiments beat observation", sets out the reasons. In the same row, write down how large the groups were, how they were compared and what has been left out. If the evidence is a percentage, decide whether it is relative or absolute. Lesson 3.3, "Percentages, risk and the base rate", shows you how to tell.

In the third row, take the four alternatives in turn and write at least one specific candidate under each. Each alternative has a question to start you off:

Confounders: what sort of person would be more likely both to join and to have the outcome? Reverse causation: could the outcome have led to the supposed cause? Selection: how were the groups chosen? Chance: how big are the groups, and how many things were compared?

The fourth row describes evidence that would make the claim more likely or less likely to be true.

The fifth row holds the verdict. It is one of three labels, followed by two or three sentences explaining your choice. Likely causal means there is experimental evidence, or strong observational evidence where the main alternatives have been checked and ruled out. Possible means the link is real and a causal story is plausible, but nobody has ruled out the alternatives. Not shown means the evidence cannot separate causation from the alternatives, or the link itself is doubtful.

A verdict of Possible does not mean the intervention is useless. It means the claim has not shown how much the intervention helps.

Daniel checks the mentoring headline

Daniel works at a company that runs a mentoring programme (Daniel and every figure in his case are invented). The HR newsletter opens with the headline "Mentoring works: mentees promoted 40 percent faster" and urges staff to apply for the next intake. Two colleagues forward it to him, and each one adds "worth doing". Daniel has nothing against mentoring. He wants to find out whether the programme makes promotion come sooner, or whether the people who sign up would have been promoted anyway.

In row 1 he writes the narrow sentence from earlier: joining the company mentoring programme causes employees to be promoted sooner than they otherwise would have been.

For row 2, he notes that the numbers come from the last three intakes. Mentees reached promotion after an average of 2.1 years. Other staff took an average of 3.5 years. Subtracting gives 3.5 - 2.1 = 1.4 years, and 1.4 / 3.5 = 40 percent, so the headline figure is relative. Staff decided for themselves whether to apply and managers decided who got in, which makes the evidence observational. The newsletter doesn't say how many mentees there were. It also doesn't say whether the two groups held similar roles and levels.

Row 3 gets three confounder entries. First, ambitious people are more likely to apply and also more likely to push for promotion. Second, a supportive manager might both nominate someone and promote them. Third, some departments promote faster than others and may also encourage mentoring. Under reverse causation, he writes that staff already in line for promotion may be encouraged to join as preparation. Under selection, he notes two things. Managers picked from the applicants and may have chosen people they already rated highly. And "other staff" includes people on contracts or in roles with no promotion path. Under chance, he writes that the intake size is unknown, and in a small intake a few quick promotions could shift the average a lot.

In row 4 he gives two tests that would settle it. One is to compare mentees with similar staff in the same roles and grades who applied but were not accepted. The other is to fill the next intake by lottery among eligible applicants. He also gives a simpler check that would still help: compare mentees with colleagues on the same grade and manager rating.

His verdict in row 5 is Possible. He gives his reasons: the link is probably real, but ambition, manager support and manager selection could account for all of it. A lottery for the next intake, or a comparison with rejected applicants, would tell him much more. He adds that none of this makes mentoring useless. The newsletter just hasn't shown how much it helps promotion.

Checking the sheet and passing it to a colleague

Your sheet is finished when it has all of these: the claim written as "X causes Y", the evidence labelled observational or experimental along with its size, at least one specific candidate under each of the four alternatives, a note of what would settle it, and a verdict backed by two or three sentences of reasons.

Other people tend to spot confounders you missed. For that reason, the activity below ends with a colleague reading your verdict.

Choose your claim now, preferably one people around you already believe.

Fill in the causal claim worksheet for one real claim, then show your verdict to a colleague and ask for a second opinion.

Course

Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).