Dark Launch: Test New Code Invisibly in Production

Why it exists

Releasing new, heavy backend features is risky. They can slow down the entire system or cause unexpected failures under real-world load. We need a way to test the performance and stability of new code with production traffic before users are affected or even aware of the feature.

The mental model

Think of it as a ghost feature. The backend logic runs for real users, triggered by their actions, but the results are never shown in the UI. It's a performance test using live traffic as the test data, completely invisible to the end-user. The system does all the work, but throws the result away before it reaches the interface.

How it works

You integrate the new backend feature into your production codebase, but with no user interface to activate it. You then modify an existing part of your application—like a checkout flow—to call this new code. The call is wrapped in a feature flag. When the flag is on, the new code executes, adding load to your system. You monitor performance, latency, and error rates. If there's a problem, you flip the flag off instantly. Once performance is acceptable, you can build the UI and reveal the feature.

When to use it

Dark launching is ideal for backend changes that enhance an existing user interaction without requiring a user's choice. Two common uses are: first, assessing the performance impact of a new service like a recommendation engine; second, verifying the correctness of a refactored algorithm by running it in parallel with the old one and comparing results, a technique called parallel running.

When not to use it

Avoid dark launching for features that require direct user interaction or choice to be tested. If you need to see how users behave with a new button, form, or workflow, you can't test that invisibly. For that, a Canary Release, where a small subset of users gets the new UI, is the appropriate strategy.

One canonical example

A retail site wants to add product recommendations to its checkout page. Before building the UI, they dark launch the recommendation engine. The checkout process is modified to call the engine for every user. The recommendations are never displayed, but the team can measure the extra milliseconds added to checkout time and the CPU load on the service. They tune the algorithm until the performance impact is negligible, then finally enable the UI for all users.