Debug intermittent pod-to-pod connectivity

WHAT THIS TESTS This evaluates structured troubleshooting across the Kubernetes networking stack and the underlying cloud substrate, and whether you can isolate intermittent faults, which are far harder than total outages because they are timing-dependent and do not reproduce on demand. The interviewer wants a hypothesis-driven method, not random restarts.

A GOOD ANSWER COVERS First scope the problem: is it pod-to-pod on the same node, cross-node, pod-to-service, or DNS? Use a debug container such as netshoot to run curl, dig, ping, and tcpdump. Confirm DNS resolution, since intermittent failures often trace to CoreDNS load or ndots search-domain expansion. Inspect Services: kube-proxy programs iptables or IPVS, and a classic cause of intermittent resets is the conntrack table filling or SNAT source-port collisions on DNAT. Check the CNI for IP address exhaustion in the pod subnet, overlay encapsulation problems, or stale routes. At the cloud layer verify security groups, NACLs, and route tables, and especially MTU: an overlay adds headers, so an MTU mismatch drops only large packets, which looks intermittent. Watch node metrics for packet drops and conntrack counts.

COMMON WRONG ANSWERS Restarting pods or the CNI without forming a hypothesis. Blaming the application when the failure is path-dependent. Ignoring DNS, conntrack, and MTU, the three most common culprits. Not distinguishing same-node from cross-node traffic, which immediately localizes the fault.

LIKELY FOLLOW-UPS How does conntrack exhaustion cause connection resets? What MTU would you set on a VXLAN overlay? How do you diagnose SNAT port exhaustion? How would CoreDNS caching or NodeLocal DNS help?

ONE CONCRETE EXAMPLE Large POST requests between nodes time out while small ones succeed. tcpdump shows fragmentation-needed ICMP messages. The overlay reduced the effective MTU below 1500, so oversized packets are silently dropped; lowering the pod MTU to match the encapsulated path resolves the intermittent failures without touching the application.