What is the difference between a primary metric and a guardrail metric?

WHAT THIS TESTS: This question tests whether you understand that experimentation is not just about proving a hypothesis but also about proving absence of harm. Interviewers want to see that you can design a decision framework with two distinct lanes: one for opportunity and one for safety. Senior candidates are expected to explain how guardrail metrics act as circuit breakers that override primary metric wins.

A GOOD ANSWER COVERS: First, define a primary metric as the pre-registered measure of success that the experiment is designed to move, such as revenue per user or conversion rate. Second, define a guardrail metric as an invariant or protective measure that should not degrade, such as page load time, error rate, or customer support tickets. Third, explain the decision rule: a result ships only if the primary metric improves statistically and no guardrail metric breaches its pre-defined threshold. Fourth, give a realistic scenario where the primary metric wins but the guardrail fails, forcing a no-ship call.

COMMON WRONG ANSWERS: A major red flag is calling guardrail metrics secondary success metrics; they are not goals to optimize but boundaries to respect. Another red flag is saying you would monitor guardrails post-launch rather than pre-registering them; this invites p-hacking and hindsight bias. Candidates who say they would ship if the primary metric is up even when a guardrail is down show poor product judgment and risk tolerance.

LIKELY FOLLOW-UPS: The interviewer may ask how many guardrail metrics you should run before multiple comparison corrections become necessary. They may ask how you handle a guardrail that dips slightly but not statistically significantly. They may also ask whether guardrails should be one-sided or two-sided, or how you prioritize guardrails when they conflict with each other.

ONE CONCRETE EXAMPLE: Imagine an e-commerce checkout flow experiment where the primary metric is checkout conversion. The team adds aggressive upsell modals and sees a plus 8 percent lift in conversion, which is statistically significant. However, the guardrail metric of page load time degrades by 300 milliseconds and the guardrail metric of mobile crash rate doubles. Even though the primary metric is up, the correct decision is to not ship because the user experience degradation and stability risk will erode trust and long-term retention, turning a short-term revenue win into a long-term liability.