You will check every statistic and chart in a real report or presentation.
A consultancy sends your company a glossy twenty-page report on hybrid work, and your head of HR wants to quote it in a proposal to cut office space. You have been asked to "sanity check the numbers". Most people would skim it, find nothing obviously wrong, and say it looks fine. This exercise gives you a way to check it properly in about thirty-five minutes, using everything from lessons 3.1 to 3.4.
Use a public report, a company presentation or an internal deck that makes a case with numbers. Industry surveys, annual reports, government publications and vendor white papers all work. Pick one with at least five statistics and at least one chart, so there is something to check, and keep it to a document you are allowed to share notes on.
Set up a sheet in a spreadsheet with one row for every number or chart in the report. Give it these columns:
The statistic or chart, copied exactly, with its page number Absolute or relative: is it a count, a rate, a percentage change, or an average, and if an average, mean or median Base given: does the report tell you the starting number, the sample size and who was in the sample Chart checks: axis starting point, time window, totals or per person, and whether it uses two axes Who is missing: anyone left out of the data who might change the answer Question for the author
The first two columns are the inventory. The rest are your checks.
Go through the report page by page and fill in a row for each statistic. Do not skip the ones that look harmless. Small numbers in footnotes are often where the important limits are hiding.
For each one, ask the questions from this module in order. Is this absolute or relative, and if relative, do I have the base from lesson 3.3, Percentages, risk and the base rate? If it is an average, is it the mean or the median, and would a few extreme values pull it, as lesson 3.2, Averages that hide the story, showed? How big was the sample, and who was in it? Who would not be in it, as lesson 3.4, Survivorship bias and missing data, asked?
For each chart, apply lesson 3.1, Charts that tell a story the data does not. Read the axis labels first. Check where the vertical axis starts on bar charts, what time window is shown and why, whether totals should be per person, and whether two different axes are making lines look connected.
Leave a cell blank only when the check does not apply. If the report does not give you what you need, write "not given". Those cells often turn into your questions.
Priya, from earlier in the module, audits the hybrid work report. All its figures here are made up for the exercise.
Row one: "78 percent of employees prefer hybrid work." It is a percentage of survey respondents. The base is given only in a footnote: 412 people who answered an online survey sent to the consultancy's newsletter subscribers. Priya notes that newsletter readers who chose to answer a survey on hybrid work are unlikely to represent all employees, and that people who left jobs over return-to-office rules may be overrepresented or missing, depending on who subscribes.
Row two: "Productivity up 22 percent at hybrid firms." Relative, with no base. Priya writes "not given" for the base and for how productivity was measured. She notes it is also a causal claim, which Module 4 deals with.
Row three: a bar chart of office use by weekday. The axis starts at 40 percent, so Tuesday's bar looks three times Friday's when the values are 58 and 46 percent. She notes the chart would need a zero baseline to compare days fairly.
Row four: "Average saving of S$1.2 million a year on office space." Priya marks it as likely a mean across firms of very different sizes, so a few large firms may be pulling it up, and notes that a median, or a saving per employee, would mean more for a company of her size.
Row five: a line chart of "hybrid adoption" starting in 2020. She notes that a window starting at the height of the pandemic would make any trend look dramatic, and asks what a longer window shows.
Once the rows are done, read down the last column and pick the three questions that would change your view most if they were answered. Write them so the author could reply in an email, without needing a meeting.
Priya's three: "How were the 412 respondents recruited, and how do their roles and industries compare with the wider workforce? How was productivity measured in the 22 percent figure, and against what comparison group? Is the S$1.2 million a mean or a median, and what is it per employee?"
None of these accuse the consultancy of anything. Each one asks for information the authors almost certainly have.
A finished audit has a row for every number and chart in the report, with "not given" wherever the report left a gap, and three questions at the bottom. Above the questions, write one sentence on how much weight the report can carry. Priya's reads: "The report shows that a self-selected group of respondents likes hybrid work, but does not give enough to support the productivity or savings figures for a company like ours."
Pick your report now, ideally one that someone at work is about to rely on, because the activity below is where you run the audit for real.
Complete the numbers audit checklist for one report and write the three questions you would send its author.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).