You will be able to explain what pretraining is, what data it uses, and what the model that comes out of it can and cannot do.
Before you read on, write down three topics you would trust an AI assistant to know well and three you would trust it less on. You will check that list against how these models learn, because where a model's knowledge comes from tells you a lot about where it will hold up.
A useful comparison is a new hire who has read most of the public internet, a huge library of books and millions of pages of code, but has never spoken to a customer. They know a huge amount, but when you ask them a question they might reply with three more questions, because that is what the FAQ pages they read looked like. A pretrained AI model is in a similar position.
Building an assistant like ChatGPT, Claude, Gemini or Copilot happens in three stages: pretraining, then fine-tuning (lesson 4.2), then learning from human feedback (lesson 4.3). This module follows all three in that order, and the first is covered here.
Pretraining is training a language model on a vast collection of text to do one thing: predict the next token. Almost all of the model's knowledge comes from this stage. It uses the guess, measure and adjust loop from lesson 2.2, applied to the next-token prediction you met in lesson 3.2.
The builders first collect the text from public web pages, books, articles, forums, reference sites and a great deal of computer code. They filter it to remove spam, duplicates and some harmful material, and what is left is still a broad slice of what people have written.
Training then runs as a loop. The model reads a passage and predicts each next token in it. It compares each prediction with the token that actually came next and adjusts its billions of dials to bring its guesses closer. It repeats this passage after passage across the whole collection.
Nobody labels anything during pretraining. The text supplies its own answers, since the correct next token is whatever the author actually wrote next. With no one needed to mark up examples by hand, it is possible to train on very large amounts of material.
Predicting the next token sounds like a narrow task, yet doing it well across that much text requires the model to absorb a lot:
Grammar, to predict the next word of a sentence. Facts, to continue "The capital of Malaysia is" correctly. Document structure, to continue a legal contract, a recipe or a Python function. Some grip on the steps of reasoning, to finish a worked example in a textbook.
Nobody teaches the model any of these directly. It picks them up because each one improves its predictions. A pretrained model ends up with broad general knowledge, the ability to write in many styles and languages, and patterns of reasoning.
It also absorbs whatever is wrong or lopsided in its text. If a myth appears online more often than its correction, the model may learn the myth. Places that are written about more are known better. For many Singapore specifics, such as CPF rules or the names of smaller neighbourhoods, there is much less text than for American topics, so the model's knowledge in those areas is thinner and less reliable.
Its knowledge also has an end date. Everything it knows stops at the date the text was collected, called the knowledge cutoff. You will test a model's cutoff yourself in lesson 4.4.
At the end of pretraining the model is very good at continuing text, and it does nothing else. It continues whatever document it thinks it is reading and has no sense that it is meant to help you.
Type "How do I renew my passport?" and it might give an answer. It might equally add more questions, such as "How long does it take? What documents do I need? Where do I collect it?", since a list of questions on a government FAQ page is a likely continuation. That matches the new hire who answers a question with three more. A pretrained model will also continue harmful or rude text as readily as polite text, because it read plenty of both.
To make the predictor answer questions, follow instructions and decline some requests, builders need the two later stages: fine-tuning in lesson 4.2 and learning from human feedback in lesson 4.3.
Pretraining at the scale of today's leading assistants needs enormous computing power running for weeks or months in large data centres, plus the work of collecting and cleaning the text. Only a handful of large companies and labs can afford it.
For this reason there are relatively few top models, known as frontier models, and many other AI products are built on top of one of them rather than trained from scratch. The later stages cost far less, so a company can take an existing pretrained model and shape it for its own purpose without paying for pretraining again.
For the activity, return to the new hire and to the list you wrote at the start. They read a broad slice of public writing, with far more on American topics than Singapore ones, some myths alongside the facts, and nothing after the date their reading stopped. Use that to predict where their knowledge will be strong and where it will be patchy.
List three things a model would probably learn from reading most of the public web, and two things it would probably get wrong because of what the web contains.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).