Transformer Preprocessing: From Text to Tensors
Transformers don't read text; they read numbers. A tokenizer is the translator, converting sentences into numerical tensors the model understands. This is the mandatory first step for any NLP task. The footgun is using a tokenizer that doesn't match the model.
Why it exists
Neural networks, including Transformers, are mathematical functions that operate on numbers, not raw text strings. To process human language, we need a consistent and reversible way to convert sentences into a numerical format that a model can comprehend. This conversion process is called preprocessing.
The mental model
Think of a tokenizer as a bilingual dictionary for humans and machines. It takes a sentence, breaks it into known words or word-pieces (tokens), and looks up the corresponding ID number for each piece in its dictionary (the vocabulary). The result is a sequence of numbers the model can process, like a machine-readable version of the original text.
How it works
The process has three main steps. First, the tokenizer splits the input string into a list of tokens. These can be words, subwords, or even individual characters, depending on the tokenizer's rules. Second, it maps each token to a unique integer using its internal vocabulary. Third, this list of integers is converted into a tensor, a multi-dimensional array that serves as the direct input to the model. This step often includes adding special tokens and padding sentences to a uniform length.
When to use it
This preprocessing step is required every time you feed text into a Transformer model. Whether you're performing classification, translation, or generation, the raw text must first be tokenized. Libraries like Hugging Face's Transformers provide an AutoTokenizer class that automatically downloads and configures the correct tokenizer for any given pretrained model from the Hub.
When not to use it
You don't use a text tokenizer when a model's input is already in a non-text format, such as image pixels for a vision model or audio waveforms for a speech model. However, the general principle of preprocessing input into a standardized tensor format is nearly universal across all deep learning domains.
One canonical example
To prepare text for the distilbert-base-uncased-finetuned-sst-2-english sentiment analysis model, you first load its specific tokenizer. You then pass your raw text to it. For example, the list of sentences ["I love this!", "I hate this!"] is passed to the tokenizer, which converts them into a dictionary containing input_ids as a PyTorch tensor, ready to be fed into the model for inference.
Interview question
What fundamental limitation necessitates the tokenization step for feeding text into a Transformer model?
- a.Tokenization is essential for compressing text data to fit within the model's memory constraints.
- b.Transformer models are mathematical functions that exclusively process numerical tensors.Correct
- c.Raw text lacks the necessary semantic embeddings that the model requires for understanding.
- d.Direct text input would lead to an unmanageable vocabulary size for the model to learn.
Why? this is the answer
The card explicitly states that "Neural networks, including Transformers, are mathematical functions that operate on numbers, not raw text strings." Thus, text must be converted into a numerical tensor format. While tokenization helps manage vocabulary and prepares for embeddings, the core reason is the model's numerical input requirement.
Just read this? Test yourself on what you have been reading.
Read the original → huggingface.co
- #llm
- #transformers
- #nlp
- #tokenizer
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on llm — each one lists the topics its interview covers.
See open roles