You will be able to adjust a probability up or down in proportion to the strength of new evidence.
In lesson 7.1, Siti put a 60 percent chance on resignations falling within six months of the new hybrid policy. Two months in, she gets new information: the monthly exit numbers for the pilot team are slightly lower than last year. A colleague says, "Great, so it's working." Another says, "One month means nothing." Siti has to decide what to do with her 60 percent. Leave it where it is? Jump to 90? Somewhere in between?
This lesson is about that question: how far to move a probability when new evidence arrives. The figures in the worked example are made up, and the calculations have been checked.
Every estimate should start somewhere sensible, and the best starting point is usually the base rate: how often this kind of thing happens in general. Lesson 3.3, Percentages, risk and the base rate, showed how ignoring base rates misleads, and lesson 5.3, Vivid stories and the planning fallacy, used them for project estimates as the outside view.
Suppose Siti's company runs a lot of projects with outside vendors, and records show that about 30 percent of them finish late. That is the base rate for "this vendor project will be late". Before she knows anything specific about a new project, 30 percent is a sensible first estimate.
Then you adjust for the specifics. If this vendor has a strong track record, she might move down a little. If the scope is unusually large, up a little. The base rate anchors the estimate in reality, and the details move it from there.
Now the vendor misses the first milestone. How much should that change the estimate?
It depends on how much more often that happens on late projects than on on-time ones. Suppose the records show that 60 percent of late projects missed their first milestone, but only 15 percent of on-time projects did. The miss is four times as common on late projects, so it is strong evidence.
Work it through with 200 past projects. At a 30 percent base rate, 60 were late and 140 were on time. Of the 60 late ones, 60 percent, or 36, missed the first milestone. Of the 140 on-time ones, 15 percent, or 21, missed it. So 57 projects missed the first milestone, and 36 of those turned out late. That is about 63 percent. One piece of strong evidence has moved Siti's estimate from 30 to about 63 percent.
Now suppose the evidence were weaker. Imagine the records showed that 50 percent of late projects and 40 percent of on-time projects missed the first milestone. A miss is only slightly more common on late projects. Out of the same 200, 30 late projects and 56 on-time projects would have missed it, so 30 of 86, about 35 percent, end up late. The estimate barely moves, from 30 to about 35.
This is the principle at work, formally known as Bayes' rule. You do not need the formula. The question to ask is: how much more likely would I be to see this evidence if my belief were true than if it were false? If the answer is "much more likely", move a lot. If it is "only slightly more likely", move a little. If it is "equally likely either way", do not move at all.
Siti's two colleagues are making opposite mistakes. "It's working" treats a single month of slightly lower exits as strong evidence, when monthly numbers bounce around anyway. "One month means nothing" treats it as no evidence at all. The honest answer is that it is weak evidence in the right direction, and her 60 percent might move to 62 or 65, not to 90.
The political scientist Philip Tetlock spent years studying how accurate people's forecasts are. In Superforecasting, written with Dan Gardner, he describes a large forecasting tournament in which a group of ordinary volunteers, whom he called superforecasters, consistently beat others at predicting world events.
One of the habits that set them apart was how they updated. They revised their forecasts often, as new information came in, and usually in small steps. They did not ignore news that cut against their view, and they did not lurch from 30 to 90 percent on a single headline. Many small, frequent adjustments kept them closer to the truth than occasional big swings.
That is a useful model at work. Rather than holding an opinion until it is overwhelmed and then reversing completely, adjust a little each time something relevant happens.
Each time you change an estimate, write a short note: the old number, the new number and the evidence that moved you. "Vendor project: 30 percent to 63 percent late. Missed first milestone; historically four times as common on late projects."
These notes do two things. They make you think about how strong the evidence really is before you move, which slows down the overreaction. And they give you a record to learn from. When you look back in lesson 8.3, Review your decisions every quarter, you will be able to see whether you tend to move too far or not far enough, and on what kind of evidence.
Something has probably happened at work since you wrote your five predictions in lesson 7.1. In the activity below you will go back to them and decide how far that new information should move each one.
Revisit the five predictions from lesson 7.1 after one new piece of information and update each probability with a reason.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).