Log Exposure, Not Just Assignment

Why it exists

Randomized assignment is only the first half of an experiment. The second half is verifying that the assigned user actually interacted with the treatment. If a user is bucketed into a new checkout flow but abandons the session before reaching checkout, including them in analysis adds noise. Exposure logging was created to draw a bright line between eligibility and actual contact with the variant, so analysts measure only the users who could have been influenced.

The mental model

Think of assignment as being handed a ticket and exposure as walking through the turnstile. You can hold a ticket to a concert and still stand outside the venue. Analyzing everyone who holds a ticket instead of everyone who entered mixes non-participants with participants and drags the signal toward zero.

How it works

When a user lands in an experiment bucket, the assignment is typically logged at the point of randomization, often near the top of the request stack. Exposure is logged separately, usually when the treatment code path actually executes or when the UI element renders. The two events are joined downstream by experiment ID and user ID. Analysts then filter to exposed users before computing metrics, or they model assignment as the instrument and exposure as the compliance stage in a causal framework.

When to use it

Use exposure logging whenever the treatment is not guaranteed to reach every assigned user. This is common in conditional flows like search result ranking, modal dialogs triggered by heuristics, or backend changes that only affect certain account tiers. It is also essential when running intent-to-treat analyses where you need both assignment and exposure to compute complier average causal effects or to debug drop-off between bucket and view.

When not to use it

Do not rely solely on exposure logging if you need to preserve the randomized intent-to-treat principle for a primary business metric like overall revenue, because conditioning on exposure can introduce selection bias if the act of becoming exposed is correlated with user intent. In those cases, assignment is the safer denominator, and exposure is used only for secondary guardrail metrics or for understanding reach.

One canonical example

A growth team tests a new upsell banner on the billing page. They assign ten million users fifty-fifty at the API gateway, but only two million ever visit billing. The team logs assignment on every request and logs exposure only when the banner component mounts. If they analyze all ten million users for subscription upgrades, the lift looks negligible because eight million users never had a chance to see the banner. Filtering to the two million exposed users reveals a clear positive effect, while comparing the exposed group against the full control group would be invalid because the control group lacks the same selection filter. The correct approach is to compare exposed treatment users against exposed control users, or to use assignment as the denominator for a conservative intent-to-treat estimate.