You will be able to spot when an extreme result was always likely to be followed by a more ordinary one.
Open your records for one result that gets tracked where you work. Monthly sales, complaint counts, error rates or your own review score are all fine. Look back over the past year and find the worst month. Write down what was changed shortly after it and how the numbers moved in the months that came next. Hold on to that example while you read, so you can test each idea below against something you know.
Every result you measure at work has two ingredients. The first is the underlying level, meaning how good the team or person usually is. The second is luck. Luck covers things like which customers happened to call in a given month, a system outage, a large order that arrived in March instead of April, or a flu spreading through the office.
When a figure is far above or far below normal, luck has probably helped push it there. A perfectly ordinary team can have an awful month when several bad things happen together. The chance of that same mix turning up again the next month is low. So the following month usually lands nearer the team's normal level, whatever anyone changes in the meantime.
This is called regression to the mean: after an extreme result that was partly down to chance, the next result is usually closer to the average. Nothing is actively pulling the numbers back. It happens for two plain reasons. What you measure has luck mixed into it, and you chose to look closely because the figure was extreme.
Organisations usually act when results are at their lowest. A bad quarter leads to a new manager. A jump in errors leads to a new process. A sales slump leads to a consultant being brought in. The action comes right after the extreme figure, so the next reading tends to be better even if the action had no effect, and the action gets the credit.
Daniel works at a company with a customer service team. March was that team's worst month on record for complaints. A new team lead joined at the start of April and brought in a fresh script and daily huddles. By May, complaints were back to normal. She was applauded at the quarterly town hall, and Daniel clapped along with everyone else. Later he asked himself whether April and May would have turned out the same if she had never arrived.
None of this proves her changes were useless. The script may well make a real difference. But the drop in complaints can't show that on its own, because some recovery was likely anyway.
You'll see the same thing in private life. Many people see a doctor when their back pain is at its worst, and a week later it has eased. People start a diet after the year's highest reading on the scale, or change how they study after their worst exam. Any of these changes might help. Each one, though, was made at the point when things were most likely to improve without it.
The effect also works in the other direction. A very good result is likely to be followed by a more ordinary one, and that can teach managers the wrong lesson about praise and criticism.
Daniel Kahneman describes a case in his book Thinking, Fast and Slow. He was teaching Israeli Air Force flight instructors about the psychology of training, and one instructor told him praise did not work. Whenever this instructor praised a cadet for an excellent manoeuvre, the next attempt was usually worse. Whenever he shouted at a cadet after a bad one, the next attempt was usually better. He concluded that criticism works and praise does not.
Kahneman explained it as regression to the mean. An excellent manoeuvre was partly luck, so the cadet's next try was likely to be closer to their usual standard whatever the instructor said. A terrible manoeuvre worked the same way in reverse. The pattern the instructor noticed was real, but he picked the wrong cause for it.
At work, this can lead a manager to give less praise and more criticism, based on what feels like solid experience. Take a salesperson who is congratulated after a record month. Their next month will probably be more ordinary. The congratulations did not make them complacent. The record month was partly luck.
Your best protection is a comparison group: similar teams, people or months that did not get the change, used to judge whether the change caused the result. Before you give a change the credit, find out how the comparison group did over the same period.
Go back to Daniel and suppose his company has three customer service teams. Only one got the new lead, and the other two also had bad Marches. Daniel can look up how those two teams did in April and May. If all three improved by similar amounts, the new lead was not what drove the improvement. If her team improved much more than the other two, that is better evidence her changes worked.
A longer record helps if you have one. Say the team had a year of steady numbers and the bad month was an unusual low. In that case, a return to the steady numbers is what you would expect.
Do the check in this order:
Find the comparison group: similar teams, people or months that did not receive the change. Look at how they did over the same period, for example how the other teams did after their own bad months. If everyone improved by similar amounts, give the change no credit. If the changed group improved much more, treat that as better evidence in its favour. Where you can, check the longer history to see whether the extreme result was an unusual dip or spike against a steady baseline. If there is nothing to compare against, conclude only that results improved and that you can't yet tell how much was due to the change, and keep measuring.
It's common to have no comparison at all. Then the only fair conclusion is the one in the last step: results got better, and you don't yet know how much of that was due to the change. Keep collecting the numbers. It is better still to set up a comparison group before the change starts, so you aren't looking for one after the result is already in. Lesson 4.2, Why experiments beat observation, showed how to build one in from the start.
Most workplaces have a story like the customer service turnaround, where a change was celebrated after a bad patch, or a slump followed a record month. In the activity below you will find an extreme result of your own, check it against the longer record, and write down what a fair comparison would need.
Find one unusually bad or unusually good result at work and check the longer record to see whether it was an extreme. Write down whether the next result moved back towards normal, and what a fair comparison would need before crediting or blaming any change.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).