Stage 2 · What happens when you write to it
Token
What is a token?
The unit the model reads and writes in. In text they are neither words nor letters: they are pieces of words, and on average a word takes a little more than one token. Images and audio are turned into tokens too.
How a model might split a sentence (illustrative; every model splits differently):
The context window fills up fast.
6 words → 9 tokens
An image to remember it
The syllables of a language the machine made up.
The misunderstanding
It reads letter by letter, like I do. It sees pieces, not letters. That's why it sometimes fails at counting how many "r"s a word has.
What changes in practice
Everything is measured in tokens: what it costs, your plan's usage limits and how much fits in the context window. Training too: today's models learn from tens of trillions of tokens, mostly English text. That's why they tend to perform worse in languages with less material, and with images or audio, than with English text.
