Try a simple example
How does AI read words?
Split one word into smaller pieces. Then change it and see what stays the same.
Break this word into pieces.
The word you can read
rereading
Invented vocabulary. These pieces and counts are for this demo, not a real model.
The worked example
Our invented vocabulary contains re, read, ing, a space, ! and café. Its longest-match rule splits reading into read + ing: two pieces. Rereading with a lowercase r becomes re + read + ing: three. Adding ! makes four. These are authored teaching counts, not a real tokenizer’s output.
A word is not always one piece.
Language models process text as tokens. A token can be a whole word, part of a word, or punctuation. In this demo, “rereading” becomes “re” + “read” + “ing”.
Use the right ruler for a real task
Word count and token count are different. When a tool has a token limit, use that tool’s tokenizer or token counter. This little demo cannot estimate its usage or cost.
Go deeper
Is this a real tokenizer?
No. This uses a hand-written vocabulary and a longest-match rule. Actual tokenizers have their own vocabularies and algorithms. For example, byte-pair encoding uses learned merge rules. Our counts are not interchangeable with theirs.
Why show spaces and capital letters?
The exact input matters to tokenization. This demo preserves spaces, punctuation and case. Its fallback rule uses Unicode code points; one visible emoji or accented character can contain more than one code point.
What happens after text is split?
A tokenizer maps its pieces to token IDs for a model to use. Our little v0, v1 and u… labels only identify this demo’s pieces. They are not IDs from Vinci or another model.
Explore the original sources
Primary references. These explain the mechanisms; the examples here are authored teaching simulations.
What would you like to try next?
Choose another small example, or return to this one whenever you need it.
See all the examples