Traffic Mirroring: Test in Production, Safely

WHY IT EXISTS: How do you know if a new service version can handle production load before you actually send it production traffic? Traffic mirroring solves this by letting you test new code with real, live requests without any risk to the user experience. It separates testing from serving.

THE MENTAL MODEL: Think of traffic mirroring as a police wiretap on your network. A proxy listens to a request, sends the original to its destination as normal, and simultaneously sends a copy to a "shadow" service for analysis. The original caller gets their response from the main service, completely unaware a copy was ever made. The shadow service's response is thrown away.

HOW IT WORKS: In a service mesh like Istio, the sidecar proxy (e.g., Envoy) intercepts a request. Based on configuration, it forwards the request to the primary service version. It then creates a copy of that request—headers and body—and sends it to the mirrored service. This is a "fire-and-forget" operation. The proxy does not wait for a response from the mirrored service, and any response it does send back is discarded. This ensures the latency of the mirrored service never affects the real user.

WHEN TO USE IT: Use mirroring to validate a new service version before a canary release. It's perfect for checking for errors, measuring performance, and comparing the behavior of v1 and v2 under identical, real-world traffic patterns. It's also useful for warming up caches on a new deployment before it starts receiving live traffic.

WHEN NOT TO USE IT: Do not use mirroring if the mirrored service performs stateful operations you don't want to duplicate. For example, mirroring a request to a payment service would charge the customer twice. The mirrored service must be either read-only or modified to use a staging database or mock external calls. Also, remember that mirroring doubles the request volume for the mirrored portion of traffic, which impacts cost and infrastructure load.

ONE CANONICAL EXAMPLE: In Istio, you define a VirtualService for myservice. The route directive sends 100% of traffic to the v1 subset. You then add a mirror field to that same route rule, pointing to the v2 subset. Now, for every request that hits myservice, Istio's proxy sends it to v1 to generate the user's response, and also sends an asynchronous, fire-and-forget copy to v2. You can then inspect the logs and metrics of v2 to see how it behaved.