What problem can a breaking API change cause during a rolling update?

WHAT THIS TESTS: This question checks whether you understand that a rolling update is a gradual replacement, not an atomic switch. Kubernetes keeps old pods running while it starts new ones, so both versions coexist behind the same Service for a window of time. The interviewer wants to see if you recognize that breaking API changes create a split-brain scenario where traffic crosses incompatible versions, and whether you know how to observe that failure in practice.

A GOOD ANSWER COVERS: First, explain that during a rolling update the ReplicaSet scales up new pods and scales down old pods incrementally, and the Service load balancer sends traffic to any ready pod regardless of version. Second, identify the specific risk: if the new code changes a request payload, response schema, internal gRPC contract, or database format, then an old pod calling a new pod or vice versa will encounter deserialization errors, HTTP 500s, or silent data corruption. Third, describe detection mechanisms in order: failing readiness or startup probes that prevent the new pod from entering the Service endpoint list; application metrics showing a spike in HTTP 5xx or latency; log aggregation catching class cast or schema exceptions; and synthetic canary requests that exercise the API boundary during the rollout. Fourth, mention that the safest fix is backward compatibility, such as additive-only schema changes or dual-write and dual-read patterns, rather than relying on detection alone.

COMMON WRONG ANSWERS: A major red flag is claiming that Kubernetes drains all old pods before new ones start, which ignores the core mechanics of a rolling update. Another weak answer is saying you would detect the problem only after the rollout completes by running manual smoke tests, because by then users have already hit errors. Saying that health checks alone prevent the issue is also wrong, because a pod can pass a liveness probe and still serve broken API responses.

LIKELY FOLLOW-UPS: The interviewer may ask how you would prevent this with deployment strategies, so be ready to contrast blue-green and canary releases against a simple rolling update. They might also ask about handling breaking database schema changes during a rollout, or how service meshes like Istio use traffic splitting to avoid cross-version calls.

ONE CONCRETE EXAMPLE: Imagine a user service running six pods on version 1.0. You deploy version 2.0 which renames a JSON field from userName to displayName. Kubernetes starts two new pods while four old pods remain. A frontend pod sends a request to the user Service, the load balancer routes it to a 2.0 pod, but the frontend still expects userName and crashes on null. You detect this because the new pods readiness probes pass, yet your metrics dashboard shows a 400 percent jump in NullPointerException logs and HTTP 500 responses from the frontend tier exactly when the new pods join the endpoint slice.

Source: geeksforgeeks.org