Design a multi-tenant ML platform with isolation, security, and cost attribution

WHAT THIS TESTS: This question evaluates whether you can translate abstract business requirements like tenant isolation, security, and cost accountability into concrete infrastructure primitives. At the senior level, interviewers care about trade-offs between hard and soft multitenancy, blast radius containment, and operational cost rather than just naming technologies.

A GOOD ANSWER COVERS: First, pick a tenancy model. Hard isolation uses dedicated clusters, node pools, or subscriptions per team for the strongest boundary but higher cost. Soft isolation shares underlying compute and uses logical separation for efficiency. Second, enforce resource isolation. In Kubernetes this means namespaces with ResourceQuotas and LimitRanges; in VMs this means separate VM scale sets or dedicated hosts with CPU and memory constraints. Third, establish security boundaries. Use network policies or NSGs to block lateral movement, enforce pod security standards or sandboxed runtimes, and scope IAM so tenant admins cannot affect other tenants. Fourth, implement cost attribution. Label every resource with tenant and project tags, aggregate metrics through a metering pipeline, and expose chargeback dashboards or quota enforcement to drive accountability.

COMMON WRONG ANSWERS: Proposing a single shared cluster with only RBAC for separation. This ignores noisy neighbor problems and lateral network movement. Suggesting physical hardware per tenant without discussing cost or utilization trade-offs. Conflating user identity with tenant identity, which breaks when teams have service accounts or external collaborators. Ignoring data plane isolation for shared storage or feature stores, which is critical in ML where datasets and model artifacts cross boundaries.

LIKELY FOLLOW-UPS: How would you handle a tenant that needs GPUs while others run CPU-only workloads? What happens when one tenant submits a training job that exhausts cluster autoscaling limits? How do you isolate shared ML metadata stores or model registries? How do you onboard a new tenant without redeploying the platform?

ONE CONCRETE EXAMPLE: An enterprise runs a central Kubernetes cluster for five data science teams. Each team gets a namespace with a ResourceQuota of 100 CPUs and 500 GB memory. A cluster autoscaler manages two node groups: CPU-only and GPU-enabled. Network policies block all cross-namespace traffic except to shared services like the central model registry, which authenticates by tenant identity. Cost allocation uses namespace labels scraped by a metering tool and fed into an internal billing API. When Team A submits a large distributed training job, it cannot evict Team B's inference pods because priority classes and preemption policies are scoped per namespace.

Source: learn.microsoft.com - Architect Multitenant Solutions on Azure - Azure Architecture Center