Tensor Parallelism: Split Layers, Not Just Models
Tensor Parallelism splits a single large model layer, like a weight matrix, across multiple GPUs to run in parallel. This is crucial for inference with models whose layers exceed a single GPU's VRAM.
Why it exists
Modern large language models have parameter counts in the tens or hundreds of billions. These parameters form weight matrices so large they often exceed the memory (VRAM) of a single GPU. To run these models for inference, we need a way to distribute not just different model layers across GPUs, but the computation within a single, massive layer.
The mental model
Think of a matrix multiplication, Y = X * W, where the weight matrix W is too big to fit on one GPU. Tensor Parallelism is like slicing that big matrix W into vertical chunks, [W1, W2], and giving one chunk to each GPU. Each GPU then works on a smaller piece of the same problem at the same time.
How it works
Using the model above, the input tensor X is sent to all GPUs. GPU 1 computes Y1 = X W1, and GPU 2 computes Y2 = X W2, simultaneously. Because the weight matrix was split column-wise, the outputs Y1 and Y2 are partial results. The final step is to gather these partial results from each GPU and concatenate them, Y = [Y1, Y2], to produce the full, correct output. This all happens within a single forward pass of one layer.
When to use it
Use Tensor Parallelism when a model's individual layers are too large to fit into a single GPU's memory. It is a foundational technique for serving state-of-the-art LLMs, enabling intra-layer parallelism that makes running huge models feasible.
When not to use it
If your model fits comfortably on a single GPU, Tensor Parallelism adds unnecessary communication overhead between GPUs, which can slow down inference. It's a solution for memory constraints, not a universal speed-up. Also, don't confuse it with pipeline parallelism, which puts different entire layers on different GPUs.
One canonical example
Serving a large model using Hugging Face's Text Generation Inference (TGI) framework is a prime example. When you launch TGI with a tensor parallelism size greater than one (e.g., for a 70B parameter model), it automatically partitions the model's weight tensors across the specified number of GPUs. For a matrix multiplication, TGI splits the weight tensor, executes the partial multiplications on each GPU, and gathers the results, all transparently. This only works for models TGI officially supports.
Interview question
What is the primary motivation for employing Tensor Parallelism in large language models?
- a.To assign distinct, complete layers of a model to different GPUs, optimizing for throughput.
- b.To achieve faster training convergence by parallelizing the gradient computations across multiple devices.
- c.To process a single, excessively large model layer by distributing its internal components across multiple GPUs.Correct
- d.To minimize inter-GPU communication by ensuring each GPU handles an entire, independent segment of the model.
Why? this is the answer
Tensor Parallelism is specifically designed to enable the execution of individual model layers that are too large to fit into a single GPU's memory by splitting the layer's components. Option A describes pipeline parallelism, which distributes entire layers, not parts of a single layer.
Just read this? Test yourself on what you have been reading.
Read the original → huggingface.co
- #llms
- #gpu
- #distributed systems
- #model serving
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on llms — each one lists the topics its interview covers.
See open roles