Design a multi-tenant model serving platform

WHAT THIS TESTS This evaluates whether you can drive down cost through resource sharing while still guaranteeing isolation, fairness, and performance, the defining tension of multi-tenancy.

A GOOD ANSWER COVERS The cost lever is sharing: instead of dedicating a GPU or endpoint per tenant, co-locate many tenants on shared serving infrastructure to raise utilization, since most tenants are idle most of the time. Then solve the hard parts. Security and data isolation: every request is authenticated and authorized to a tenant, models and data are partitioned so no tenant can access another's, and you use namespaces, separate keys, and network controls. Resource isolation: enforce per-tenant quotas, rate limits, and scheduling priorities so a heavy or abusive tenant cannot monopolize compute and degrade others, the noisy-neighbor problem. Performance: autoscale shared pools, route requests efficiently, keep hot models loaded, and offer dedicated capacity or higher tiers for tenants with strict latency SLAs. A common design is multiple small tenants sharing a pool while large tenants get isolated capacity.

COMMON WRONG ANSWERS Ignoring data isolation, risking cross-tenant leakage. Giving each tenant fully dedicated infrastructure, which abandons the cost goal. Forgetting noisy neighbors, so one tenant starves the rest. No per-tenant quotas or metering. No plan for cold-start of rarely used models.

LIKELY FOLLOW-UPS How do you stop one tenant's traffic from harming others? How do you isolate models in memory? How do you meter and bill per tenant? When do you move a tenant to dedicated capacity?

ONE CONCRETE EXAMPLE A platform serves hundreds of small customer models from a shared GPU pool on Kubernetes, loading models on demand and keeping popular ones warm. Each request carries a tenant identity checked against scoped authorization; namespaces and resource quotas cap each tenant's CPU, GPU, and request rate. A few high-volume enterprise tenants are pinned to dedicated node pools to guarantee latency, while the long tail shares capacity to keep costs low.