A key metric dropped 15%. How do you investigate?

What's really being asked

This question assesses your ability to apply a systematic debugging framework to a business problem, not just a technical one. The interviewer is looking for a structured, prioritized process. They want to see if you can move from a broad symptom (metric drop) to a specific root cause by correlating data from disparate systems: product analytics, application logs, infrastructure metrics, and deployment pipelines. It tests your ability to think like a system owner, not just a coder.

The full answer

A strong answer outlines a process that moves from broad to narrow. First, validate the data itself: is the 15% drop real, or is the analytics pipeline broken? Check the raw event volume, look for processing errors in the ETL jobs, and query the underlying database. Second, correlate the drop's timing with recent changes. Check deployment history, feature flag toggles, infrastructure changes, and even external events like marketing campaigns or holidays. Third, segment the metric. Is the drop uniform across all users, or is it concentrated in a specific segment (e.g., Android users, users in Brazil, new users)? This helps narrow the search space dramatically. Fourth, form and test a hypothesis based on the evidence.

The mistakes people make

The most common red flag is jumping to a conclusion without a process. For example, immediately saying "I'd look at the last commit" or "It's probably a bug in the new feature." This shows a lack of systematic thinking. Another mistake is focusing only on application code and ignoring the possibility of a broken data pipeline, a misconfigured A/B test, or an infrastructure issue. A weak answer is a disorganized list of things to check; a strong answer is a prioritized, ordered plan.

What usually comes next

"Okay, you've found the drop is isolated to Android users in Germany. What next?" (Tests ability to drill down). "You suspect a recent deployment is the cause, but the CI/CD logs show no changes. What are other possibilities?" (Tests creative problem-solving and considering non-obvious causes). "How would you communicate your findings to a Product Manager?" (Tests cross-functional communication).

A concrete example

"I'd first check our analytics dashboard's health. Let's say our ETL jobs run hourly. I'd check the logs for the last 24 jobs in Airflow or our data workflow tool. If they look healthy, I'd query the raw event table in Snowflake. Is the volume of 'product_interaction' events down 15%, or is the final aggregated metric table just wrong? If the raw volume is down, I'll pull deployment logs from Spinnaker and feature flag changes from LaunchDarkly that correlate with the drop's start time. If a deploy lines up, I'd start there."