Ask what data is missing from a sample before you trust its conclusion.
When confident advice comes only from people who succeeded, the people who did the same thing and failed are missing from the evidence. A success story can be real and still mislead you, because the failures are not shown beside it.
At a networking event, a speaker tells the room how she built a successful bubble tea chain. She quit her bank job with six months of savings, went all in and never had a backup plan. Her biggest applause comes for the line "If I'd kept one foot in the door, I'd never have made it."
Priya, sitting near the back, wonders how many people quit bank jobs to open bubble tea shops in the same years, and what happened to them. Her question points at the people missing from the speaker's advice: those who made the same move and failed. You rarely hear from them because nobody invites them to speak, writes about them or follows their accounts.
This is survivorship bias: drawing conclusions from the cases that made it through some filter, while ignoring the ones that did not. Most success advice has this shape. You hear from people who succeeded, and rarely from people who did the same thing and failed.
The speaker's story shows that going all in can work. It cannot show how often it works. Keeping a backup plan could even work better on average, and the stage would still be full of all-in successes, because theirs are the bold bets that happened to pay off. The same filter sits behind stories of founders who dropped out of university, investors who held one stock for twenty years, and colleagues who were promoted after switching teams.
Lesson 1.2, "Not all evidence weighs the same", showed that a single story tells you little about how often something happens. Survivorship bias adds a second problem: a story can also mislead you about direction, meaning which choice is actually the better one.
Abraham Wald was a statistician who worked with a research group advising the United States military during the Second World War. In the usual telling of his best-known problem, analysts mapped the bullet holes on bombers returning from missions. The holes clustered on the wings and on the body of the plane, called the fuselage. The obvious proposal was to add armour where the holes were.
Wald said this was backwards, because the analysts were studying only the planes that came back. A plane hit in the wings could still return, which is why returning planes showed so many wing hits. Planes hit in the engines mostly did not return, so engine hits were rare in the sample. The areas with few holes on returning planes were the areas where a hit brought a plane down, and those were the areas that needed armour.
The popular version simplifies what Wald actually wrote, but the reasoning still holds. The sample held only survivors, and the survivors differed from the lost planes in exactly the way that mattered.
Survivorship bias also turns up in ordinary workplace data, well away from famous success stories. Suppose a gym reports that 90 percent of its members are happy with the classes, a made-up figure. The survey misses people who already left, and they are often the dissatisfied ones. The result is real, but it describes only those who chose to stay. It cannot explain why others left, which is often the most important question.
The same problem appears when a company interviews only current staff about its culture, when a course provider reports results only from students who completed the course, and when a fund manager shows the track record only of funds that still exist. In each case the group being measured has already been sorted by the very thing you care about.
The fix is to seek out the missing group. Exit interviews, surveys of cancelled customers, or the full list of funds a manager has run, including the closed ones, can tell a very different story. People who tried and failed are harder to find, and their stories carry the most information.
Missing data does not always matter. If you survey current users about a new button colour, the absence of past users probably makes little difference to the result. Missing data matters a great deal when whatever removed people from the data is related to what is being measured, such as dissatisfaction, failure or death.
You can work through any piece of evidence in the same way:
Whenever you see evidence offered for a conclusion, ask who is not in this data, and why. Ask whether they would be different in a way that changes the answer. If whatever removed them is related to what is being measured, such as dissatisfaction, failure or death, treat the gap as serious. Go and find the missing group through exit interviews, surveys of cancelled customers, or full records that include closed or failed cases. If the gap is unrelated to the question, as with the button colour, the missing group matters little.
Back at the networking event, Priya does not need to tell the speaker she is wrong. She can hold the advice more loosely, and if she ever considers a similar leap, she can look for people who tried it and did not succeed before she acts on it.
Think of a piece of success advice you have heard more than once, from a book, a podcast or a senior colleague. In the activity below you will write down who would be missing from the evidence behind it.
Pick one piece of success advice you have heard. List the people who would be missing from the evidence behind it.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).