Design a multi-model serving architecture for infrequently used models

WHAT THIS TESTS: This question tests your ability to design cost-efficient, low-latency inference infrastructure for sparse traffic patterns. The interviewer cares about trade-offs between resource utilization and isolation, your understanding of caching and eviction strategies, and whether you can mitigate cold starts without provisioning dedicated hardware for every model.

A GOOD ANSWER COVERS: First, a shared compute pool that dynamically loads and unloads models based on demand rather than maintaining always-on endpoints per model. Second, a tiered storage strategy where model weights live in cheap object storage but are cached on fast local NVMe or SSD when active, using an LRU or frequency-based eviction policy. Third, concrete cold-start mitigation such as keeping a base container or runtime warm, predictive preloading based on time-of-day patterns, or using an async queue with a 202 Accepted response for uncached models so the caller is not blocked during load. Fourth, resource isolation and contention management through per-model memory caps, CPU or GPU slice limits, concurrency throttling, and priority classes so one large model cannot evict everything else or starve neighbors. Fifth, horizontal scaling of the shared pool based on aggregate load, plus health checks and graceful draining to avoid dropping requests during model swaps.

COMMON WRONG ANSWERS: Proposing dedicated endpoints or Kubernetes Deployments for each model ignores the cost constraint and fails the prompt. Suggesting a single monolithic container that loads all models at startup ignores memory and startup time reality. Saying simply scale to zero without addressing the latency penalty of reloading multi-gigabyte weights from remote storage is incomplete. Ignoring resource isolation is a red flag because in production one misbehaving model can crash the shared node.

LIKELY FOLLOW-UPS: How would you handle models that need incompatible CUDA versions or runtimes? What happens if two models together exceed available GPU memory? How do you roll out a new model version without restarting the endpoint? What is your target P99 latency and how does it change for cached versus uncached models? How do you charge back cost to individual teams when compute is shared?

ONE CONCRETE EXAMPLE: Imagine a GPU instance with 16 GB of VRAM serving a catalog of two hundred models. Each model takes 2 GB when loaded and ten seconds to fetch from remote storage. You pin the top five models in memory permanently, use an LRU cache for the next three, and leave the rest on disk. When a request arrives for an uncached model, you evict the oldest cached model and load the new one in a background thread while returning a 202 status to the client. You enforce a max concurrency of four requests per model and reserve one GPU slice for critical models. This keeps steady-state cost at one instance instead of two hundred, while bounding cold-start latency for the tail.