Design a multi-tenant GPU serving system for hundreds of fine-tuned models

WHAT THIS TESTS: This question tests your ability to design a multi-tenant GPU inference platform that maximizes utilization across hundreds of fine-tuned models while respecting hard memory constraints and latency service level objectives. You must demonstrate command of tiered memory hierarchies, dynamic loading tradeoffs, and scheduling strategies that prevent one tenant's cold start from starving others.

A GOOD ANSWER COVERS: First, tiered model storage and staging. Keep hot model weights in GPU memory, warm weights in host CPU RAM, and cold weights on fast local NVMe or network-attached storage so you are not limited by GPU VRAM capacity. Second, predictive pre-warming and demand loading. Use usage-pattern forecasting or time-of-day heuristics to copy likely-next models from storage into CPU RAM or directly into GPU memory before the first request arrives. Third, disaggregated inference architecture. Split prefill and decode phases across different GPUs or nodes so that loading a model into a prefill pool does not block decode capacity, allowing independent optimization and faster time-to-first-token. Fourth, SLO-aware scheduling and queueing. Hold or route incoming cold-model requests to a staging queue while weights load, or redirect to a shadow instance, rather than failing or accepting unbounded latency. Fifth, memory extension via caching. Transfer KV cache and model shards across heterogeneous memory and storage tiers to free GPU memory for active weights while preserving user context.

COMMON WRONG ANSWERS: A major red flag is proposing to keep all hundreds of models resident in GPU memory simultaneously, which is physically impossible at scale. Another is ignoring the latency cost of transferring multi-gigabyte weights over PCIe or NVLink and simply stating that the system will load on demand with no buffering or queueing strategy. A third is conflating model loading with KV cache management, failing to recognize that even after weights load, attention state must be handled separately.

LIKELY FOLLOW-UPS: The interviewer may ask how you would handle a thundering herd when a popular model suddenly goes cold to hot, how to shard a single large model across multiple GPUs while still sharing the node with other fine-tuned models, or what metrics you would expose to an autoscaler to decide when to evict versus replicate a model. They may also probe how you bound tail latency when a request arrives exactly as a model unload is in progress.

ONE CONCRETE EXAMPLE: NVIDIA Dynamo addresses this exact problem through several components relevant to this design. Its SLO Planner monitors capacity and prefill activity in multi-node deployments, adjusting GPU resources to consistently meet latency targets. The KV Block Manager transfers KV cache across memory hierarchies, freeing GPU memory while maintaining user experience. NIXL, the low-latency communication library, accelerates transfer of KV cache between GPUs and across heterogeneous memory types. Dynamo also supports disaggregated serving, splitting prefill and decode phases across nodes for independent optimization, and extends GPU memory through data caching to cost-effective storage tiers. Independent benchmarks show that GB300 NVL72 combined with NVIDIA Dynamo improves mixture-of-experts model throughput by up to 50x compared to Hopper-based systems, leveraging high-speed NVLink for low-latency expert communication.

Source: developer.nvidia.com