Tokenization and Input Embeddings in LLMs

WHY IT EXISTS: Neural networks are mathematical functions that consume tensors of floating-point numbers. They have no native notion of letters, words, or grammar. To make language modelable by matrix multiplication, we need a lossless bridge that converts variable-length text into fixed-shape numerical input. That bridge is the combination of tokenization and input embeddings.

THE MENTAL MODEL: Think of tokenization as creating the alphabet of the machine, and input embeddings as giving each character in that alphabet a personality. The tokenizer defines the vocabulary, typically tens of thousands of entries built from subword frequencies. The embedding layer is simply a lookup table where each row is a learned vector. When the model sees token ID 4819, it retrieves row 4819, a vector of perhaps a few thousand dimensions, and that vector becomes the initial representation fed into the transformer stack.

HOW IT WORKS: A tokenizer first breaks text into chunks using an algorithm like Byte Pair Encoding or SentencePiece. It starts with characters and iteratively merges the most frequent adjacent pairs into new symbols until the vocabulary reaches its target size, often fifty thousand to a few hundred thousand entries. Each chunk gets an integer index. The embedding layer stores a weight matrix of shape vocabulary size by embedding dimension. During the forward pass, the model uses these indices to gather the corresponding rows, producing a sequence of vectors. Positional encodings or rotary embeddings are then added so the model knows the order of the sequence.

WHEN TO USE IT: This pipeline is mandatory for every transformer-based generative model, including large language models, vision transformers that process patch tokens, and multimodal systems that interleave text and image tokens. Anytime you see a model consuming discrete symbols and producing continuous representations, this two-step conversion is happening under the hood.

WHEN NOT TO USE IT: You do not need a separate tokenizer and embedding layer when working with inherently continuous input, such as raw audio waveforms or pixel values, which can be projected directly via linear layers. Also, if you are fine-tuning a pretrained model, you generally should not replace the tokenizer or reinitialize embeddings unless you are extending the vocabulary, because the model's knowledge is tightly coupled to the specific token IDs it was trained on.

ONE CANONICAL EXAMPLE: In the GPT family, the text "unhappiness" might be tokenized into ["un", "happ", "iness"] rather than one word. Each subword gets an ID, say [347, 9821, 445], and the embedding layer fetches three vectors. The transformer then processes these three vectors as the initial state for predicting the next token. If you change the tokenizer, the same string would map to different IDs and the pretrained model would produce garbage, which is why the tokenizer config is shipped with every model checkpoint.