Isolating the design system's velocity impact

WHAT THIS TESTS This tests statistical literacy and intellectual honesty: can you separate correlation from causation and build a defensible attribution rather than overclaiming because timing lines up.

A GOOD ANSWER COVERS First, concede the stakeholder has a point: velocity is multi-causal, so naive before-and-after attribution is weak. Then propose methods to isolate the effect. Use a comparison or control group: compare teams that adopted versus similar teams that did not over the same period, which cancels out company-wide factors. Apply a difference-in-differences design to estimate the adoption effect net of the shared trend. Use the same team's longer baseline before and after, controlling for confounders like team size changes, project complexity, hiring, and tooling shifts. Strengthen the causal story with mechanism evidence: component reuse counts, reduced UI implementation and rework time, fewer design-related defects, and developer-reported time saved, which show a plausible pathway, not just a correlation. Present effect sizes with uncertainty and avoid claiming the entire fifteen percent. Where possible, a staggered rollout acts as a natural experiment.

COMMON WRONG ANSWERS Claiming the full fifteen percent is the design system because the timing coincides, which is pure correlation. Ignoring confounders entirely. No comparison group, so any company-wide trend is misattributed. Presenting point estimates with no uncertainty.

LIKELY FOLLOW-UPS What confounders worry you most? How does difference-in-differences work here? How would a staggered rollout help?

ONE CONCRETE EXAMPLE You compare five adopting teams against five matched non-adopting teams over two quarters using difference-in-differences; the adopters show an eight-point velocity gain net of the shared trend, and component-reuse and rework data explain the mechanism, so you defend roughly half the observed fifteen percent as attributable with stated uncertainty.