Explain what tokens are and predict when a text will use more tokens than you expect.
Text you send to an AI assistant is chopped into small chunks and converted into numbers before the model does any work on it, so the model has no direct view of individual letters. Knowing this explains what assistants cost to run, where their limits come from and several of their stranger errors.
For a long time, ask many assistants how many letter r's appear in "strawberry" and you would often get a wrong count. Those same tools could summarise a fifty-page report or write a calm reply to a furious customer. A primary school child can count the letters in a word, which makes the failure look ridiculous until you learn that the assistant was never shown the letters.
Language models work in tokens, which are chunks of text. Plenty of tokens are whole everyday words, such as "the", "and" or "meeting". A long or unusual word gets broken into several tokens. Spaces and punctuation marks normally count as tokens, or get attached to one.
The chunking happens in a tokenizer, a program that runs on its own before the model receives your input. It works from a fixed vocabulary of chunks. To build that vocabulary, its makers went through an enormous amount of training text and found the strings of characters that turned up most often. A word that appears constantly gets its own token, and a less common word has to be assembled from smaller pieces. So "appointment" could be a single token, while "Kallang" could come out as two or three pieces, perhaps "K", "all" and "ang". Treat that split as an illustration only. Every tokenizer divides text in its own way, and different assistants use different tokenizers.
A model can only calculate, and it cannot calculate with words. The next stage therefore swaps each token for a long list of numbers that the model learned during training. It helps to picture each list as a set of coordinates marking where the token sits. Tokens that get used in similar ways land near each other, so "clinic" is closer to "hospital" than it is to "durian".
Your text reaches the model in three stages. The tokenizer chops it into tokens from its fixed vocabulary. Each token is exchanged for its learned list of numbers. The model then does arithmetic on those lists, and from there on arithmetic is all it does.
Unless a single letter happens to be a token by itself, the model has no view of letters. "Strawberry" may reach it as two or three chunks, each one a list of numbers, and none of those numbers states directly how many r's the word has.
English made up much of the training text, so common English words are usually cheap at one word, one token. Several kinds of text use more:
rare words and names, which often break into several tokens; many Singapore place names, Malay and Hokkien terms, and brand names fall here Chinese, Tamil, Thai and other scripts, which often need more tokens to say the same thing, depending on the tokenizer numbers, which get cut into chunks of digits, so a long bank account number or an NRIC-style string turns into a run of separate pieces computer code, whose symbols, brackets and spacing usually take more tokens than ordinary prose
Token counts have two practical effects. Assistants set their limits in tokens, so a Chinese document, or one packed with figures, reaches the limit sooner than plain English of similar length. This limit is the context window, and Module 5 covers it. Services that bill by usage also generally count tokens, so token-heavy text costs more to process. If your company builds products on these models, you'll run into this.
Counting letters is difficult because the letters sit inside chunks the model can't look into. It only answers correctly if training taught it which letters each chunk holds, and its knowledge of that is uneven. A useful workaround is to ask the model to write the word out letter by letter first, and only then count or answer. Once spelled out, each letter becomes a separate token the model can see.
Long numbers carry a similar risk. A sixteen-digit card number may be divided into uneven groups of digits, and the model can slip when copying it, reversing it or doing sums with it. When a figure from an assistant actually matters, for example a total on a bank statement or a reference number, compare it with the source document yourself.
Rhymes, anagrams and word puzzles go wrong for the same underlying reason. They rely on letters and sounds, and the model handles chunks of meaning. Newer assistants often do better on these, sometimes by writing the word out in parts and sometimes by running a small program to do the counting, a method lesson 7.3 covers. The cause itself is still there, so keep asking for words to be spelled out and keep checking figures that matter.
Seeing your own text cut into tokens makes all of this concrete in a couple of minutes. Several free online tokenizers show the split in colour as you type, and most people are surprised by which of their everyday sentences turn out to be the expensive ones.
Paste three short texts into a free online tokenizer: one in English, one with numbers and one in another language you know. Note which one used the most tokens.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).