Multi-tenant isolation with a monitoring exception

WHAT THIS TESTS This is a design question combining strict tenant isolation with a single controlled exception for observability. It tests layering of NetworkPolicies and use of namespaceSelector.

A GOOD ANSWER COVERS Start from default-deny: in every tenant namespace apply a policy selecting all pods with policyTypes Ingress and no allow rules, which drops all inbound traffic. Then add the allowances you actually want. Typically you permit intra-namespace traffic with an ingress rule whose from uses a podSelector or an empty namespaceSelector matching the same namespace, so a tenant's own pods can talk. Critically, cross-tenant traffic stays denied because no rule permits other tenant namespaces. For monitoring, label the monitoring namespace, for example team=monitoring, and in each tenant namespace add one ingress rule whose from has a namespaceSelector matching that label, scoped to the metrics port such as TCP 9090 or the app's metrics port. Now Prometheus pods in monitoring can scrape every tenant, but tenants remain isolated from one another. Egress policies can further lock down outbound traffic, and you must allow DNS to kube-system.

COMMON WRONG ANSWERS Opening all cross-namespace ingress to make scraping work destroys isolation. Using pod IP CIDR blocks instead of namespaceSelector is brittle because pod IPs churn. Forgetting to label the monitoring namespace so the selector matches nothing, silently blocking scrapes.

LIKELY FOLLOW-UPS How to template these policies per namespace. Why egress DNS must be allowed. How Cilium or Calico enforce this and add L7 controls.

ONE CONCRETE EXAMPLE Tenant tenant-a has default-deny, an allow-same-namespace rule, and an allow-from namespace team=monitoring rule on port 9090. Prometheus scrapes tenant-a and tenant-b, but tenant-b pods cannot reach tenant-a at all.