You will run a set of small tests that expose token-by-token generation and sampling in a real assistant.
You have now read about three mechanisms: text cut into tokens, answers built one predicted token at a time, and sampling that makes each attempt a little different. This exercise lets you catch each of them in the act, in whichever assistant you normally use. It takes about twenty-five minutes and needs nothing except the assistant and the results table below this lesson.
A few ground rules make the results cleaner. Start each test in a new chat, so earlier messages do not influence the answer. If your assistant has a memory or personalisation feature, note whether it is on. Copy each prompt exactly, and write down what came back before you try anything else.
Pick a long word with a repeated letter, such as "accommodation" or "Mississippi". Ask the assistant how many times a particular letter appears in it, for example how many m's are in "accommodation". Write down the answer and whether it is correct. Count it yourself on paper first.
Then, in a new chat, ask it to spell the word one letter at a time, each on a new line, and only then count the letter, so you can put the two results side by side.
The mechanism here is tokens, from lesson 3.1. In the first attempt the letters are hidden inside chunks. In the second, spelling the word out turns each letter into its own token that the model can see. Newer assistants often get both right, sometimes because they quietly run a small program to count. If yours does, note that too.
Ask for ten name ideas for something specific, such as a home-based bakery in Punggol that sells kueh, and copy the list somewhere safe before you open two more fresh chats and ask exactly the same thing.
Count how many names appear in more than one list. Some overlap is normal, because the most likely names will come up repeatedly. But you should see plenty of differences too. That is sampling, from lesson 3.3: each attempt is a separate draw from the same probabilities.
Pick a local event from the last week or two that you know the facts of: a new hawker centre opening, a transport announcement, a result in a sport you follow. Ask the assistant a specific question about it, without telling it the answer.
Watch for one of three things. It answers from its own knowledge, which may be confidently wrong or out of date. It says it does not know or cannot see recent events. Or it searches the web and cites sources. Record which one happened.
This test points at prediction, from lesson 3.2. Without search, the model can only continue your text with what is likely given its training, and an event after its training data was collected was never in that data. A likely-sounding answer is all it can produce. Module 4 looks at this edge directly in lesson 4.4, Find the edges of what a model knows.
Now go through your table and, for each test, write which mechanism explains what you saw: tokens, prediction or sampling. Some results have more than one cause, and saying so is fine. The habit you are building is to stop treating odd behaviour as random and ask which part of the machinery produced it.
Here is how Marcus, an admin executive at a clinic in Tampines, filled in his table. His results are an example and yours will differ by assistant and by day.
Letters: asked how many m's are in "accommodation", it said 2, which is right. Asked how many s's are in "Mississippi", it said 3, which is wrong. Spelled out letter by letter first, it counted 4 correctly. Mechanism: tokens. Names: 30 names across three lists, with 4 appearing twice and none appearing three times. Mechanism: sampling. Recent event: asked about a hawker centre that had opened ten days earlier. With search on, it found a news article and got the opening date right. In a second chat where he asked it not to search, it described a different hawker centre with a similar name. Mechanism: prediction, with search filling the gap.
What stayed with Marcus was the third test, because the made-up answer was written in exactly the same calm tone as the correct one.
Open the results table now and run the first test before reading anything else, so that what you know from this lesson does not shape how you word the prompt.
Run the four tests and fill in a table with the prompt, what happened and which mechanism, tokens, prediction or sampling, explains the result.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).