You will be able to ask what data is missing from a sample before trusting its conclusion.
At a networking event, a speaker tells the room how she built a successful bubble tea chain. She quit her bank job with six months of savings, went all in and never had a backup plan, and she gets her biggest applause for the line "If I'd kept one foot in the door, I'd never have made it." Priya, sitting near the back, finds herself wondering how many people quit their bank jobs to open bubble tea shops in the same years, and what happened to them.
Her question points to a gap that sits behind a lot of confident advice. The speaker's story is real. What is missing is everyone who did the same thing and is not on a stage.
The best-known version of this problem comes from the Second World War. The statistician Abraham Wald worked with a research group advising the United States military. The story usually told goes like this. Analysts looked at bombers returning from missions and mapped where they had been hit. The holes clustered on the wings and body. The natural idea was to add armour where the holes were.
Wald pointed out that this was backwards. The planes being studied were the ones that came back. A plane hit in the wings could return, which is why the returning planes showed so many wing hits. Planes hit in the engines mostly did not return, so engine hits were rare in the sample. So the places with few holes on the returning planes were the places where a hit brought a plane down, and those were the places that needed the armour.
The popular telling simplifies what Wald actually wrote, but the reasoning holds. The sample was made only of survivors, and the survivors were different from the planes that were lost in exactly the way that mattered. This is survivorship bias: drawing conclusions from the cases that made it through some filter, while ignoring the ones that did not.
Most success advice has the same shape as the bomber data. You hear from people who did something and succeeded. You rarely hear from people who did the same thing and failed, because nobody invites them to speak, writes about them or follows their accounts.
The bubble tea speaker went all in with no backup and succeeded. That tells you going all in can work. It cannot tell you how often it works, because the people who went all in and lost their savings are not in the room. It is even possible that keeping a backup plan works better on average, and the stage would still be full of people who went all in, because they are the ones whose bold bets happened to pay off.
The same goes for founders who dropped out of university, investors who held one stock for twenty years, and colleagues who got promoted after switching teams. These are real stories, drawn from groups whose failures were filtered out before you ever heard about them. Lesson 1.2, Not all evidence weighs the same, said a single story tells you little about how often something happens. Survivorship bias is the reason it can mislead you about direction too.
Survivorship bias is not only about famous success stories. It shows up in ordinary workplace data.
Suppose a gym chain surveys its members and, in this made-up example, finds that 90 percent are happy with the classes. The members who were unhappy with the classes have mostly already left, so they are not in the survey. The result is real, but it describes people who chose to stay. It cannot tell the gym why others left, which is probably the question it most needs answered.
A company that only interviews current staff about culture, a course provider that reports results from students who completed the course, or a fund manager who shows the track record of funds that still exist all face the same problem. In each case the group being measured has already been sorted by the thing you care about.
The fix is to go and find the missing group. Exit interviews, surveys of cancelled customers, or the full list of funds a manager has run, including the closed ones, can tell a very different story.
The habit from this lesson is one question, asked every time you see evidence for a conclusion: who is not in this data, and why? Then a follow-up: would they be different in a way that changes the answer?
Sometimes the answer is no. A survey of current users about a new button colour probably does not suffer much because past users are missing. But when the thing that removed people from the data is related to what you are measuring, like dissatisfaction, failure or death, the missing group matters a great deal.
Priya does not need to tell the bubble tea speaker she is wrong. She can simply hold the advice more loosely and, if she ever considers a similar leap, look for people who tried it and did not succeed. They are harder to find, and they are the ones whose stories carry the most information.
Think of a piece of success advice you have heard more than once, from a book, a podcast or a senior colleague. In the activity below you will write down who would be missing from the evidence behind it.
Pick one piece of success advice you have heard and list who would be missing from the evidence behind it.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).