Handle interaction effects on a shared page

What's really being asked

This probes whether you understand that experiments on a shared surface are not guaranteed independent and that their combination can mislead. The interviewer wants the concept of interaction effects, the risk they pose, and a concrete strategy spanning mutual exclusion and monitored orthogonality.

The full answer

An interaction effect occurs when the effect of one experiment depends on which variant of another a user received. On a shared page, two changes can clash visually or behaviorally, for example one experiment enlarges a banner and another moves the call-to-action under where the banner now sits, so users in both variants behave differently than either change alone predicts. The measured average effect of each experiment then mixes in the other's influence, biasing both readouts. Strategies: if you expect interaction, because the experiments touch overlapping elements, put them in the same layer so assignment is mutually exclusive and no user is ever exposed to both. If interaction is unlikely and you want full traffic, run them orthogonally but actively monitor: analyze each experiment's effect separately within each variant of the other, and check whether the effects differ meaningfully. If they do, you have an interaction and should rerun with mutual exclusion or analyze the combined cells. Detecting interactions reliably needs more sample, so plan power accordingly.

The mistakes people make

Assuming concurrent same-page experiments are always independent. Never checking for interactions, so a confound goes unnoticed. Making every experiment mutually exclusive by default, wasting traffic when interaction is implausible. Pooling results without slicing by the other experiment's variant, which hides the interaction.

What usually comes next

How do you statistically detect an interaction? A factorial analysis with an interaction term, or comparing effects across the other's variants. When is mutual exclusion worth the cost? When elements overlap or stakes are high. How does power change when you must estimate interactions?

A concrete example

Two checkout experiments run orthogonally: one changes the button color, another adds a trust badge near the button. Sliced analysis shows the color effect is positive only when the badge is present and flat otherwise, an interaction. The team rebuilds the test with both changes in one mutually exclusive layer, or analyzes the four combined cells directly, rather than trusting the confounded marginal effects.