Check every statistic and chart in a real report or presentation.
Priya's head of HR plans to cut office space and wants to support the proposal with a consultancy's report on hybrid work. You met Priya earlier in the module. She has been given the glossy, twenty-page report and asked to "sanity check the numbers".
Most people asked to do this give the report a quick read. If no obvious errors stand out, they reply that it looks fine. A structured audit does the job properly. It takes about thirty-five minutes and uses the checks from Lessons 3.1 to 3.4. At the end you have three things: a log of every statistic and chart, three questions you can email to the author, and one sentence stating how much weight the report can carry.
Choose a document that makes its case with numbers. It can be a public report, a company presentation or a deck that circulates inside your organisation. Industry surveys, annual reports, government publications and vendor white papers are all good choices. The report must contain at least five statistics and at least one chart, and you must have permission to share your notes on it.
Your log is a spreadsheet, and each number or chart gets its own row. Set it up with six columns:
Column 1: the statistic or chart, in the report's exact words, with the page it appears on. Column 2: whether it is absolute or relative, and what kind of number it is (a count, a rate, a percentage change or an average). For an average, record whether it is a mean or a median. Column 3: base given. This records whether the report states the starting number, how many people were sampled and who they were. Column 4: chart checks. These cover where the axis begins, the time window, whether the chart shows totals or per-person figures, and whether it uses two axes. Column 5: who is missing. This means anyone left out of the data whose inclusion could change the answer. Column 6: the question you would put to the author.
Columns 1 and 2 make up the inventory, which records what the report claims and what kind of number each claim is. Columns 3 to 6 are the checks.
Go through the report one page at a time. Log every statistic and chart you find, even ones that look harmless. Read the footnotes too, because a small number at the bottom of a page often shows an important limit of the data.
Ask four questions about each statistic, in this order:
First, is the figure absolute or relative? If it is relative, look for the base (Lesson 3.3, Percentages, risk and the base rate).
Second, if the figure is an average, is it a mean or a median? Could a few extreme values pull it up or down (Lesson 3.2, Averages that hide the story)?
Third, how large was the sample, and who was in it?
Fourth, who could never have ended up in the sample (Lesson 3.4, Survivorship bias and missing data)?
For charts, use the checks from Lesson 3.1, Charts that tell a story the data does not. Start with the axis labels. On a bar chart, note where the vertical axis starts. Note the time window and think about why the author chose it. Consider whether totals should be shown per person instead. If a chart puts two lines on two different axes, check whether the layout makes them look connected.
Leave a cell blank only when the check does not apply to that row. If the check applies but the report does not give you the information, write "not given". Many of your questions for the author will come from these cells.
Priya's first row was the headline claim, "78 percent of employees prefer hybrid work" (an invented figure, like every number in her audit). She logged it as a percentage of survey respondents. The base appeared only in a footnote: 412 people who answered an online survey sent to the consultancy's newsletter subscribers. These were newsletter readers who chose to answer a survey about hybrid work, so Priya noted that they are unlikely to represent all employees. She also noted that people who had left jobs over return-to-office rules could be overrepresented or missing, depending on who subscribes to the newsletter.
Her second row was "Productivity up 22 percent at hybrid firms". It is a relative figure with no base. She wrote "not given" for the base and also for how productivity was measured. She flagged it as a causal claim, which is covered in Module 4.
The third row was a bar chart of office use by weekday. Its vertical axis started at 40 percent. Tuesday's value was 58 percent and Friday's was 46 percent. The part of each bar you can see above the 40 percent line is 58 - 40 = 18 for Tuesday and 46 - 40 = 6 for Friday. As a result, Tuesday's bar looks three times as tall as Friday's, even though the real values are 58 and 46 percent. Priya recorded the truncated axis. Cutting an axis like this exaggerates the difference between bars, and a fair comparison needs the axis to start at zero.
Row 4 was "Average saving of S$1.2 million a year on office space". Priya judged that this was probably a mean taken across firms of very different sizes, so a few large firms could push it up. For a company the size of hers, a median or a per-employee saving would be more useful.
Row 5 was a line chart of "hybrid adoption" starting in 2020. A time window that starts at the height of the pandemic makes any trend look dramatic. Her question for this row asked what a longer window would show.
When every row is complete, read down column 6. Choose the three questions whose answers would most change your view. Word each one so the author can reply by email without a meeting. Avoid anything that sounds like an accusation, and ask only for information the authors almost certainly have.
Priya chose these three questions:
How were the 412 respondents recruited, and how do their roles and industries compare with the wider workforce?
How was productivity measured for the 22 percent figure, and what comparison group was it measured against?
Is the S$1.2 million a mean or a median, and what does it work out to per employee?
Above the three questions, write a one-sentence verdict. Priya's verdict said that the report shows a self-selected group of respondents likes hybrid work, but does not provide enough to support the productivity or savings figures for a company like theirs.
A finished audit has a row for every number and chart, "not given" wherever the report left a gap, three questions at the bottom and the verdict sentence directly above them.
Pick your report now, ideally one that someone at work is about to rely on, because the activity below is where you run the audit for real.
Fill in the numbers audit checklist for one report, then write the three questions you would send its author.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).