Design a multi-tenant GPU serving system for hundreds of fine-tuned models
Tests GPU memory tradeoffs versus cold-start latency in multi-tenant serving. Strong answers propose tiered CPU staging, predictive pre-warming, and disaggregated prefill and decode. Red flag: keeping all models GPU-resident or ignoring transfer overhead.
What's really being asked
This question tests your ability to design a multi-tenant GPU inference platform that maximizes utilization across hundreds of fine-tuned models while respecting hard memory constraints and latency service level objectives. You must demonstrate command of tiered memory hierarchies, dynamic loading tradeoffs, and scheduling strategies that prevent one tenant's cold start from starving others.
The full answer
First, tiered model storage and staging. Keep hot model weights in GPU memory, warm weights in host CPU RAM, and cold weights on fast local NVMe or network-attached storage so you are not limited by GPU VRAM capacity. Second, predictive pre-warming and demand loading. Use usage-pattern forecasting or time-of-day heuristics to copy likely-next models from storage into CPU RAM or directly into GPU memory before the first request arrives. Third, disaggregated inference architecture. Split prefill and decode phases across different GPUs or nodes so that loading a model into a prefill pool does not block decode capacity, allowing independent optimization and faster time-to-first-token. Fourth, SLO-aware scheduling and queueing. Hold or route incoming cold-model requests to a staging queue while weights load, or redirect to a shadow instance, rather than failing or accepting unbounded latency. Fifth, memory extension via caching. Transfer KV cache and model shards across heterogeneous memory and storage tiers to free GPU memory for active weights while preserving user context.
The mistakes people make
A major red flag is proposing to keep all hundreds of models resident in GPU memory simultaneously, which is physically impossible at scale. Another is ignoring the latency cost of transferring multi-gigabyte weights over PCIe or NVLink and simply stating that the system will load on demand with no buffering or queueing strategy. A third is conflating model loading with KV cache management, failing to recognize that even after weights load, attention state must be handled separately.
What usually comes next
The interviewer may ask how you would handle a thundering herd when a popular model suddenly goes cold to hot, how to shard a single large model across multiple GPUs while still sharing the node with other fine-tuned models, or what metrics you would expose to an autoscaler to decide when to evict versus replicate a model. They may also probe how you bound tail latency when a request arrives exactly as a model unload is in progress.
A concrete example
NVIDIA Dynamo addresses this exact problem through several components relevant to this design. Its SLO Planner monitors capacity and prefill activity in multi-node deployments, adjusting GPU resources to consistently meet latency targets. The KV Block Manager transfers KV cache across memory hierarchies, freeing GPU memory while maintaining user experience. NIXL, the low-latency communication library, accelerates transfer of KV cache between GPUs and across heterogeneous memory types. Dynamo also supports disaggregated serving, splitting prefill and decode phases across nodes for independent optimization, and extends GPU memory through data caching to cost-effective storage tiers. Independent benchmarks show that GB300 NVL72 combined with NVIDIA Dynamo improves mixture-of-experts model throughput by up to 50x compared to Hopper-based systems, leveraging high-speed NVLink for low-latency expert communication.
Interview question
Which strategy best balances GPU memory constraints and latency SLOs when serving hundreds of fine-tuned models in a multi-tenant system?
- a.Stream weights on demand from storage directly into GPU memory, relying on fast interconnects to hide transfer latency without request buffering
- b.Unify model weight loading and KV cache transfers into a single sequential load from cold storage to avoid scheduling complexity
- c.Maintain tiered weight staging with predictive pre-warming into CPU RAM, disaggregate prefill and decode pools, and use SLO-aware queueing for cold startsCorrect
- d.Keep all fine-tuned model weights permanently resident in GPU memory while offloading only the KV cache to host RAM
Why? this is the answer
Option C is correct because it combines tiered memory hierarchies, predictive pre-warming, and disaggregated inference to directly address the memory-latency tradeoff. Option D is a tempting distractor because permanently keeping hundreds of models GPU-resident is physically impossible at scale and violates the core memory constraint.
Just read this? Test yourself on what you have been reading.
Read the original → developer.nvidia.com
- #mlops
- #gpu inference
- #model serving
- #distributed systems
- #memory management
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on mlops — each one lists the topics its interview covers.
See open roles