Tokens and vocabulary-size tradeoffs
Tokenization fundamentals.
A token is a subword unit; larger vocab shortens sequences but bloats the embedding matrix, smaller vocab generalizes but lengthens sequences.
What's really being asked
Whether you know tokens are subword units and can reason about how vocabulary size trades sequence length against parameter count and coverage.
The full answer
A token is the atomic unit an LLM reads and predicts, drawn from a fixed vocabulary learned by a subword algorithm such as byte-pair encoding, WordPiece, or Unigram. Tokens are often subwords: common words may be a single token, while rare or compound words split into several pieces, and this is what lets a finite vocabulary cover open text including new words. On vocabulary size, a larger vocabulary represents the same text in fewer tokens, which shortens sequences, reduces the quadratic attention cost per document, and lets more context fit in the window. The downside is the embedding table and the final output projection scale with vocabulary size, adding parameters and memory, and very rare tokens get few training updates so their embeddings are poorly learned. A smaller vocabulary keeps those matrices compact and gives each token more training signal, but the same text becomes more tokens, raising compute and consuming context. The choice also interacts with multilingual coverage, since more languages and scripts demand more vocabulary to avoid fragmenting non-English text.
The mistakes people make
A token is just a word. Bigger vocabulary is always better, or always worse. Ignoring the embedding and output softmax growth. Forgetting that small vocabularies inflate sequence length. Overlooking multilingual and rare-token effects.
What usually comes next
How does BPE build its vocabulary? Why does multilingual support push vocab larger? How does tokenization affect inference cost and pricing? What problems arise from over-fragmenting non-English text or numbers?
A concrete example
The word unbelievable might be one token in a large vocabulary but split into un, believe, able in a small one. A document that is 1000 tokens under the large vocabulary could become 1400 tokens under the small one, costing more compute and context, while the large vocabulary pays for that with a bigger embedding matrix and some rarely seen, undertrained tokens.
Interview question
What is the main downside of choosing a much larger tokenizer vocabulary for an LLM?
- a.The model can no longer represent rare or out-of-vocabulary words
- b.The embedding and output projection matrices grow, and rare tokens receive too few training updatesCorrect
- c.Sequences become longer, raising attention cost per document
- d.It prevents the tokenizer from handling multiple languages
Why? this is the answer
A larger vocabulary shortens sequences but inflates the embedding and softmax parameters and leaves rare tokens undertrained. Larger vocab makes sequences shorter, not longer, and subword schemes still cover rare words.
Just read this? Test yourself on what you have been reading.
- #llm
- #tokenization
- #bpe
- #vocabulary
- #pretraining
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on llm — each one lists the topics its interview covers.
See open roles