What is the difference between a primary metric and a guardrail metric?
Tests whether you distinguish success criteria from safety checks in experiments. A strong answer defines primary metrics as the target outcome, guardrails as protective thresholds, and gives a concrete scenario where a primary lift does not justify shipping…
What's really being asked
This question tests whether you understand that experimentation is not just about proving a hypothesis but also about proving absence of harm. Interviewers want to see that you can design a decision framework with two distinct lanes: one for opportunity and one for safety. Senior candidates are expected to explain how guardrail metrics act as circuit breakers that override primary metric wins.
The full answer
First, define a primary metric as the pre-registered measure of success that the experiment is designed to move, such as revenue per user or conversion rate. Second, define a guardrail metric as an invariant or protective measure that should not degrade, such as page load time, error rate, or customer support tickets. Third, explain the decision rule: a result ships only if the primary metric improves statistically and no guardrail metric breaches its pre-defined threshold. Fourth, give a realistic scenario where the primary metric wins but the guardrail fails, forcing a no-ship call.
The mistakes people make
A major red flag is calling guardrail metrics secondary success metrics; they are not goals to optimize but boundaries to respect. Another red flag is saying you would monitor guardrails post-launch rather than pre-registering them; this invites p-hacking and hindsight bias. Candidates who say they would ship if the primary metric is up even when a guardrail is down show poor product judgment and risk tolerance.
What usually comes next
The interviewer may ask how many guardrail metrics you should run before multiple comparison corrections become necessary. They may ask how you handle a guardrail that dips slightly but not statistically significantly. They may also ask whether guardrails should be one-sided or two-sided, or how you prioritize guardrails when they conflict with each other.
A concrete example
Imagine an e-commerce checkout flow experiment where the primary metric is checkout conversion. The team adds aggressive upsell modals and sees a plus 8 percent lift in conversion, which is statistically significant. However, the guardrail metric of page load time degrades by 300 milliseconds and the guardrail metric of mobile crash rate doubles. Even though the primary metric is up, the correct decision is to not ship because the user experience degradation and stability risk will erode trust and long-term retention, turning a short-term revenue win into a long-term liability.
Interview question
An experiment shows a significant increase in its primary metric, but page load time degrades past a pre-defined threshold. What is the correct product decision?
- a.Do not ship because the threshold breach acts as a circuit breaker on the primary winCorrect
- b.Lower the page load threshold and continue running the experiment
- c.Reclassify page load time as a secondary metric since conversion improved significantly
- d.Ship and observe page load time in production before making a final call
Why? this is the answer
Guardrail metrics are pre-registered safety thresholds that override primary-metric wins, so the correct call is to not ship. Shipping and monitoring post-launch is a common anti-pattern because it invites hindsight bias and fails to protect users from known harm.
Just read this? Test yourself on what you have been reading.
Read the original → statsig.com
- #experimentation
- #metrics
- #a/b testing
- #product judgment
- #statistics
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on experimentation — each one lists the topics its interview covers.
See open roles