Rank pieces of evidence by how much they can support a claim.
Pick one piece of evidence you were shown at work recently, such as a testimonial, a usage figure or a colleague's complaint, and write down who produced it. Pieces that look equally convincing on the page can carry very different weight, and where a piece came from is the first clue to how much.
Lesson 1.1, "Claim, evidence and the link nobody says out loud", showed you how to separate a claim from what it rests on. The next job is judging how much each piece of evidence can carry once you have it in front of you.
Wei Ling runs operations for a small logistics firm in Tuas (an invented example). On Tuesday her manager, Kumar, forwards her a sales deck for a new warehouse scanning system. Page three has a client quote: "Our picking errors dropped overnight." Page five says the system is used by over 200 warehouses. That afternoon, a friend in a WhatsApp group says her company tried something similar and hated it, and mentions that the scanners struggled in a cold room.
By Wednesday, Wei Ling has three pieces of evidence pointing in different directions, and she has to tell Kumar what she thinks.
A single story, whether it's a testimonial, an anecdote or a complaint, shows that something happened at least once. It says very little about how often it happens. Stories are easy to remember and repeat, which is why sales decks and group chats are full of them.
The client quote on page three is a story, and the vendor picked it from all its clients. If the product worked for one warehouse in ten, the deck would still show that one. The friend's complaint has the same weakness in reverse: one bad rollout could be caused by the system, the training, the timing or the company.
Stories are still useful, because a story can point to a mechanism you had not thought of, which you can then check. The friend's comment about scanners in a cold room is a specific worry that can be tested. A story cannot tell you how likely a good or bad outcome is for you.
Counts and surveys can carry more weight than stories, but only under two conditions: size and representativeness. More cases beat fewer, because in a small group a couple of unusual cases can swing the whole result. Ten customers tell you more than one, and five hundred tell you more than ten.
Representativeness matters more than size. A representative sample is one where the people counted look like the wider group the claim is about, in the ways that matter for the question. A survey of five hundred customers who chose to fill in a feedback form is large but not representative. Very happy and very annoyed people fill in forms, and the quiet middle does not.
The "over 200 warehouses" figure has a related gap. A count of current users says nothing about those who tried the system and left, a problem you will meet again in lesson 3.4, "Survivorship bias and missing data". When you see a user count, ask what kind of users they are, how long they have used the product, and how many tried it and left.
Someone produced every piece of evidence you see. Before you weigh it, ask three plain questions. Who produced this? What do they gain if I believe it? What would they have left out?
A source that has something to gain is not necessarily wrong. The vendor gains a sale, so its deck will show the best clients, the best results and the most flattering way to count them. People inside your own company can have a stake too. If the scanning system was Kumar's idea, he will notice good news first. Wei Ling's friend has no stake in the sale, which counts in her favour, but she may hold a grudge against a project that made her weekends miserable.
The third question usually does the most work. If a vendor reports error rates after a rollout but not before, the comparison is missing. If it reports satisfaction without saying how many people responded, the size is missing. The missing pieces are often more telling than what is on the page.
Watch for repetition as well. Suppose Wei Ling searches online and finds "errors dropped overnight" on a trade website, a LinkedIn post and a blog, and all three quote the same press release. That is one source seen three times, and it adds nothing to the original quote.
Evidence gets stronger when separate sources, working independently, reach the same answer. If two warehouses Wei Ling contacted directly, with no link to the vendor, both reported fewer errors, that would count for much more, because each could have been wrong in its own way and neither was.
A rough ladder, from weakest to strongest, has five rungs:
a single story or testimonial a handful of hand-picked cases a count or survey of a group that may not be representative a larger, fairer sample several independent sources that agree
A piece of evidence can move up or down this ladder depending on who produced it and what they left out. A vendor's user count with no figure for people who left is weaker than it first looks.
When you have evidence in front of you, work through it in this order. Treat each single story as proof that something happened once, and note any mechanism it suggests that you can check. For counts and surveys, check the size, then whether the group looks like the people the claim is about.
Ask the three questions of every source, and look for missing comparisons, such as before and after, and missing sizes, such as how many responded or how many left. Check whether sources that seem separate trace back to one origin, and look for independent sources with no link to whoever is making the claim. Then place each piece on the ladder and turn what you find into specific next steps for the person making the decision.
Wei Ling replies to Kumar on Wednesday afternoon. She tells him the deck has one hand-picked quote and a user count with no information on users who left. Her friend's story is a single case, but it raises a specific worry about scanners in cold rooms that they can check. She suggests calling two warehouses of similar size that the vendor did not choose, and asking the vendor for before-and-after error rates at three sites. The reply takes her ten minutes and gives Kumar something he can act on.
Think of a decision your own team made recently, such as a new tool, a change of supplier or a hire. What evidence was on the table when you made it? In the activity below you will list it and put each piece in order of how much weight it could really carry.
List the evidence behind one recent decision your team made. Rank each piece from weakest to strongest and give a reason for each.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).