You will be able to move a probability up or down in proportion to how strong the new evidence is.
When new information arrives, most people handle it in one of two ways, and both make forecasts worse. Some swing their whole view on a single piece of news. Others hold on to their first guess until the evidence is overwhelming, then flip completely. A better approach is to start your estimate in a sensible place and move it in proportion to how strong each new piece of evidence is.
Siti works at a company that recently introduced a hybrid work policy. In lesson 7.1 she estimated a 60 percent chance (an example figure) that resignations would fall within six months of the policy starting. The policy has now run for two months, and she sees that the pilot team's monthly exit numbers are slightly lower than last year. One colleague says this means the policy is working. Another says one month of data means nothing. Siti has to decide whether to stay at 60, jump to 90, or land somewhere in between. To make that call well, she first needs a method she can apply to any estimate.
The best starting point for an estimate is usually the base rate, which is how often this kind of thing happens in general. Siti's company runs many projects with outside vendors, and about 30 percent of them finish late. When a new vendor project begins, her starting estimate that it will finish late is 30 percent. Lesson 3.3, Percentages, risk and the base rate, showed how ignoring base rates misleads people. Lesson 5.3, Vivid stories and the planning fallacy, used base rates for project estimates as the outside view.
The base rate ties your estimate to what usually happens. The specific details of the case then adjust it up or down, usually by a little. A vendor with a strong track record might justify moving the late estimate down a little. An unusually large scope might justify moving it up a little.
Once you have a starting number, new evidence will keep arriving. How far a piece of evidence should move your estimate depends on how much more often that evidence appears when your belief is true than when it is false. Strong evidence moves an estimate a lot. Weak evidence moves it a little. Evidence that is equally likely whether your belief is true or false should not move the estimate at all.
For each new piece of evidence, ask yourself: "How much more likely would I be to see this if my belief were true than if it were false?" Then classify it as strong, weak, or equally likely either way, and move your estimate by a matching amount. This principle is formally called Bayes' rule. You do not need the formula to use it.
Siti's vendor misses its first milestone. She looks at the company's history and finds that 60 percent of late projects missed the first milestone, compared with 15 percent of on-time projects. Dividing 60 by 15 gives 4, so a missed first milestone is four times as common on late projects as on on-time ones. That makes it strong evidence that this project will finish late.
To see how far strong evidence should move an estimate, you can run a quick check using a round sample. The steps are:
Pick a round sample size. Split it into groups using the base rate. Count how many in each group show the evidence. Take the share of evidence-showing cases that match the outcome. That share is your updated estimate.
Siti picks 200 past projects. At a 30 percent base rate, 60 of them were late and 140 were on time. Of the 60 late projects, 60 percent, or 36, missed the first milestone. Of the 140 on-time projects, 15 percent, or 21, missed it. That gives 36 + 21 = 57 projects that missed the first milestone, and 36 of those 57 were late, which is about 63 percent. Siti's estimate moves from 30 to about 63 percent.
Suppose instead that the company's history had looked different: 50 percent of late projects and 40 percent of on-time projects missed the first milestone. Using the same 200 projects, 60 late and 140 on time, 50 percent of 60 gives 30 late projects that missed it, and 40 percent of 140 gives 56 on-time projects that missed it. That is 30 + 56 = 86 misses in total, and 30 of 86 is about 35 percent. In this case the estimate barely moves, from 30 to about 35. A missed first milestone can be strong or weak evidence, depending on how much more common it is in one group than the other.
Back to the hybrid policy. Monthly exit numbers go up and down naturally, so one month of slightly lower figures is weak evidence. It does point the right way, though, so it justifies a small move. Siti should move from 60 to 62 or 65, and she should not go to 90. Her first colleague makes the overreacting mistake by treating weak evidence as strong. Her second colleague makes the ignoring mistake by treating weak evidence as no evidence at all.
Philip Tetlock, a political scientist, spent years studying how accurate people's forecasts are. In the book Superforecasting, written with Dan Gardner, he describes a large forecasting tournament in which a group of ordinary volunteers, the superforecasters, consistently beat others at predicting world events. What set them apart was that they revised their forecasts often as new information arrived, usually in small steps. They did not ignore news that cut against their view, and they did not swing from 30 to 90 percent on a single headline. Their many small, frequent adjustments kept them closer to the truth than occasional big swings would have.
At work, this means adjusting your opinion a little each time something relevant happens. Avoid holding your view until the evidence overwhelms you and then reversing completely.
Every time an estimate changes, write a short note with the old number, the new number and the evidence that caused the move. Siti's note for the vendor project reads: "Vendor project: 30 percent to 63 percent late. Missed first milestone; historically four times as common on late projects."
Writing the note makes you judge how strong the evidence is before you move, which slows down overreaction. Over time, the notes also build a record of whether you tend to move too far or not far enough, and on which kinds of evidence. You will review your update notes in lesson 8.3, Review your decisions every quarter.
Now go back to the five predictions you wrote in lesson 7.1. For each one, list any new information that has come in since you wrote it. For each piece of information, ask how much more likely you would be to see it if your prediction were true than if it were false, and decide whether it is strong evidence, weak evidence, or equally likely either way. Move each estimate by a matching amount, or leave it where it is, and write a note for each one with the old number, the new number and the evidence that moved you.
Go back to your five predictions from lesson 7.1. After one new piece of information, update each probability and write down the reason.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).