Describe how a language model builds a full answer by predicting the next token again and again.
Pick up your phone, open a new message and type "Your polyclinic appointment is on". Look at the row of suggested words above the keys. Your phone might offer "Monday", "the" and "Tuesday" (an invented example, since every phone learns its own). If you tap one suggestion after another, a sentence forms, though it is usually a dull one.
A large language model is built on the same basic idea, pushed a long way further. It predicts the next piece of text, adds it, and predicts again. It differs from keyboard suggestions in how much it has learned and how much of the text it considers before each guess. With that much more to go on, it can draft a report or explain a clause in your tenancy agreement.
Before producing anything, the model reads everything written up to that point. That covers the question you typed, any instructions the app slipped in, and the part of its own reply it has already written. All of it is broken into tokens, the pieces of text the model reads and writes, and passed through the model's billions of learned settings.
What comes out is a score for each token in its vocabulary, meaning the complete set of tokens it is able to score and produce. The scores become probabilities, one for each token, showing how likely it is to come next. Each step yields a spread of likelihoods learned from the model's training text, and no single token is marked as the answer.
Go back to the polyclinic sentence. Suppose the model rates "the" at 25 percent, "Monday" at 18 percent, "Tuesday" at 15 percent and "Friday" at 12 percent. Every remaining token in the vocabulary gets a sliver, "banana" included. These figures are made up to illustrate the point, but real outputs have this same shape: a handful of strong options and a very long list of tokens that each get a tiny probability.
From that spread, one token is picked. It is not always the top-scoring one, and Lesson 3.3 covers how the pick is made. The picked token goes onto the end of the text, and the model runs the whole process again on the slightly longer text.
Each new token shifts the odds for the one after it. If "the" is picked, the text reads "Your polyclinic appointment is on the", and dates and numbers now sit near the top of the list. Put in order, one pass of the loop goes:
read all the text so far as tokens, including your question, the app's instructions and the reply so far pass those tokens through the learned settings to get a score for every token in the vocabulary turn the scores into probabilities for what comes next pick one token from that spread, not always the likeliest (see Lesson 3.3) add the picked token to the end of the text go back to the start with the longer text
It keeps going until the model outputs a special end-of-answer token, which means "end of answer", or until it reaches a length limit. A reply of three paragraphs takes a few hundred passes. When an assistant's reply appears on your screen in pieces, you are seeing this loop run at close to real time.
The model has no finished answer sitting out of sight that it then types out for you. Every token depends on what came before it, and a token stays once it is written. The model has no way to rewrite its first sentence after seeing where the paragraph ended up.
Researchers who study the inside of these models have found that models do some planning and look ahead a little. When composing a line that rhymes, a model can decide on the rhyming word before it writes the words that lead to it. It still commits to one token at a time, though, and at no point does it read back over the full answer and correct it. If an assistant starts with "Yes, you can claim that", the rest of the reply has to agree with that yes, even when the honest answer would have been no.
Some assistants have a thinking or reasoning mode. In this mode the model writes out working notes first and the answer second. Underneath, it is still next-token prediction. The notes let the model work the problem through on the page before it commits to the section you actually read.
Calling the model "only autocomplete" and expecting nothing from it but smooth filler undersells it. Predicting the next token well, across billions of examples of human writing, forced the model to pick up grammar, facts that come up often, the way arguments are put together, the way code fits together, and the way a summary relates to the text it summarises.
Give it a meeting transcript and ask for a summary, and the likeliest next tokens are summarising ones, since that pattern turned up often in its training data. Ask it for the GST on a S$480 invoice (an invented example), and the likely tokens follow the working a person would set down. Prediction is the method it uses, and the skill comes from everything it had to learn in order to predict well.
The failures you will study in Module 6 come from the same mechanism. The model writes what is likely, and what is likely is usually true, though not every time. A regulation that sounds plausible can be a likely string of tokens whether or not it actually exists.
You can get a feel for this without any software. Playing the model yourself for a few steps, with nothing more than a pen, shows how much each early choice decides what follows.
Write the first five words of a sentence. List three words that could come next and give each a rough probability. Then note how your choice changes the rest of the sentence.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).