Tokens and vocabulary-size tradeoffs

WHAT THIS TESTS Whether you know tokens are subword units and can reason about how vocabulary size trades sequence length against parameter count and coverage.

A GOOD ANSWER COVERS A token is the atomic unit an LLM reads and predicts, drawn from a fixed vocabulary learned by a subword algorithm such as byte-pair encoding, WordPiece, or Unigram. Tokens are often subwords: common words may be a single token, while rare or compound words split into several pieces, and this is what lets a finite vocabulary cover open text including new words. On vocabulary size, a larger vocabulary represents the same text in fewer tokens, which shortens sequences, reduces the quadratic attention cost per document, and lets more context fit in the window. The downside is the embedding table and the final output projection scale with vocabulary size, adding parameters and memory, and very rare tokens get few training updates so their embeddings are poorly learned. A smaller vocabulary keeps those matrices compact and gives each token more training signal, but the same text becomes more tokens, raising compute and consuming context. The choice also interacts with multilingual coverage, since more languages and scripts demand more vocabulary to avoid fragmenting non-English text.

COMMON WRONG ANSWERS A token is just a word. Bigger vocabulary is always better, or always worse. Ignoring the embedding and output softmax growth. Forgetting that small vocabularies inflate sequence length. Overlooking multilingual and rare-token effects.

LIKELY FOLLOW-UPS How does BPE build its vocabulary? Why does multilingual support push vocab larger? How does tokenization affect inference cost and pricing? What problems arise from over-fragmenting non-English text or numbers?

ONE CONCRETE EXAMPLE The word unbelievable might be one token in a large vocabulary but split into un, believe, able in a small one. A document that is 1000 tokens under the large vocabulary could become 1400 tokens under the small one, costing more compute and context, while the large vocabulary pays for that with a bigger embedding matrix and some rarely seen, undertrained tokens.