Name the main ways training data limits a model and spot them in a real product.
An AI model only knows what its training examples showed it. That limits what it can do in four main ways (rare cases, label errors, past decisions and the date the data stops), and each one points to where a product you use is most likely to get things wrong.
Picture a voice assistant that gets a request right when one person asks but cannot make sense of that person's grandmother saying the same words. The phone, the app and the request are identical in both cases. What differs is who is speaking, and how many people who sound like her were in the recordings the model was trained on. You saw in lesson 2.2 that a model learns by adjusting its dials until they fit its examples. It follows that whatever is missing from those examples is also missing from the model.
Training nudges a model's dials each time it sees an example, so the situations it meets most often get the most nudges and the best results. Uncommon situations get far fewer, and the dials settle around what is typical. Speech recognition makes this easy to see. If most of the training audio came from American or British speakers, the model will usually do worse on any accent that was scarce in that audio. In Singapore you notice it when voice typing trips on Singlish particles, mangles Malay or Hokkien place names, or loses track of an older speaker's English. The model has no intention of leaving anyone out; it simply had fewer recordings of those voices to learn from.
Images and medicine have the same weakness: photos shot in unusual lighting, skin tones that were under-represented in an image dataset, and scans from a type of machine the model was never shown. The working rule is to expect poorer results for any group or situation that was rare in the data, and to expect the model to carry on without any sign that it has gone beyond what it learned.
Many models learn from labelled examples, meaning training examples that a person has tagged with the right answer. A spam filter learns from emails tagged spam, a scan reader from X-rays tagged as showing a fracture, a screening tool from CVs tagged shortlisted. Whoever does the tagging can get it wrong, can disagree with a colleague, and brings their own assumptions to the job.
Imagine a team sorting customer complaints who marked angry-sounding messages urgent and politely worded ones routine. A model trained on their work picks up the pattern that tone signals urgency, and a courteous email describing a serious problem lands at the bottom of the pile. There is no fault in the model. It reproduced what the labels taught it, the staff's assumptions included.
Past decisions are one particular kind of label. Suppose a company gives a CV screening model ten years of its own hiring records, each tagged hired or not hired. What the model learns to predict is who this company tended to hire before. If earlier hiring leaned towards graduates of certain schools, or quietly skipped older applicants and people who had taken a career break to care for a parent, the model absorbs that pattern as well and then applies it quickly to every new CV that arrives.
Reuters reported in 2018 that Amazon had dropped an experimental recruiting tool after it marked down CVs containing the word "women's", for example in a women's chess club. The tool had learned from years of applications that came mostly from men.
Builders have ways to correct for this. They can compare results across groups, remove or rebalance biased examples, and test before launch, but none of these steps happens on its own. Left alone, a model trained on history keeps repeating it until someone deliberately intervenes. An HR officer should understand this before relying on any tool that ranks applicants, and should remember that Singapore's Tripartite Guidelines on Fair Employment Practices still apply to decisions made with that tool.
Training data is gathered up to a certain point and then frozen. Anything after that point is unknown to the model unless it is given fresh information while in use. Take a model that predicts polyclinic no-shows from records gathered before a new booking system launched. It has no idea the new system exists, and its predictions may drift as patients change their habits. A language model whose text stops before a rule changed will explain the old rule with complete confidence. Module 4 comes back to this limit under the name knowledge cutoff, and in lesson 4.4 you test it against recent Singapore changes.
Work through these four questions in order:
Who or what was rare in its training data? Who made the labels, and what might they have got wrong? Was it trained on past decisions, and were those decisions fair? How old is its data?
Vendors seldom say much about their training data, so you will rarely get full answers. You can usually make a reasonable guess from what the product does and who built it, and that guess shows you where to keep the closest eye on its results.
Pick a tool you know well from your own week, since you already know who uses it and where it has let someone down, and answer the four questions for it.
Pick one AI feature you use. Write down who or what might be missing or rare in its training data, and how that could show up in its results.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).