Pitfalls of 'Conversion Rate' as a North Star Metric

WHAT THIS TESTS: This question assesses senior-level product sense and systems thinking. The interviewer isn't testing if you know what 'conversion rate' is; they're testing if you understand that local optimization can cause global problems. They want to see if you can anticipate business trade-offs (e.g., conversion vs. profitability) and technical/UX degradations, and then instrument a system to detect them.

A GOOD ANSWER COVERS: First, acknowledge the value of conversion rate as a primary indicator, but immediately pivot to its lack of context. Second, identify specific, concrete pitfalls. For example, a simplified funnel might remove upsell opportunities, thus decreasing Average Order Value (AOV), or remove information, leading to higher return rates. Third, propose specific counter-metrics to measure these trade-offs directly, such as AOV, items per cart, and return rate. Fourth, propose guardrail metrics to protect system health and user experience. These are often technical, like page load time, client-side error rates, or API latency for the payment gateway.

COMMON WRONG ANSWERS: One major red flag is treating conversion rate as infallible and only discussing its benefits. This shows a lack of critical thinking. Another is suggesting vague counter-metrics like 'user happiness' without a concrete way to measure it (e.g., a post-purchase NPS survey, change in support ticket volume). A senior engineer is also expected to think about system health; forgetting technical guardrails like latency and error rates is a common mistake. Finally, confusing counter-metrics (which measure a direct business trade-off) with guardrail metrics (which protect against broader degradation) shows a less nuanced understanding.

LIKELY FOLLOW-UPS: Expect questions like: "How would you decide if a 5% increase in conversion is worth a 10% decrease in AOV?" which tests your decision-making framework. Or, "What if you can't measure return rate in your 2-week experiment window? What proxy could you use?" which tests your creativity with data. You might also be asked to describe the dashboard you'd build to monitor the experiment.

ONE CONCRETE EXAMPLE: A team redesigns a checkout page to be simpler, removing a 'You might also like' module. An A/B test shows the new design increases conversion rate by 3%. This seems like a win. However, the counter-metric 'Average Order Value' (AOV) drops by 15%, from 120 to 102. The overall revenue per visitor has decreased (Old: 120 * 10% CR = 12. New: 102 * 10.3% CR = 10.51). Furthermore, a guardrail metric shows that customer support tickets related to 'finding a specific accessory' have increased by 25%. Despite the conversion lift, the experiment is a failure due to its negative impact on revenue and support load.