Investigate a 20% drop in a key revenue metric
This tests your ability to lead a high-pressure investigation. A great answer confirms the drop, traces data from dashboard to source, and differentiates bugs from business trends. A red flag is jumping to conclusions without a systematic, layered approach.
What's really being asked
This question tests leadership and process, not just technical debugging. The interviewer wants to see if you can manage a high-stakes, ambiguous situation. They're testing your ability to think systematically from the top down (dashboard to source), communicate effectively with stakeholders (product, finance, data), and differentiate between a technical bug and a genuine business event. It's a test of your systems thinking applied to a socio-technical system, assessing your seniority beyond just coding.
The full answer
A strong answer outlines a clear, phased investigation. First, contain and communicate: acknowledge the severity, inform stakeholders of an investigation, and time-box the initial effort to 1-2 hours in a dedicated channel or war room. Second, validate the drop: do not trust the dashboard. Cross-reference with an orthogonal source, like raw payment processor reports or a simple query against the production database's transactions table. Third, trace the data lineage: start at the dashboard query and work backward. Check for recent code changes, then move upstream to the data warehouse table, the ETL/ELT job, and its logs, metrics, and traces. Look for schema changes or increased nulls. Fourth, differentiate bug vs. business: if no technical fault is found, segment the data. Is the drop uniform across all user segments, regions, or platforms? A drop isolated to Android users in Germany after a release points to a bug. A uniform drop across all cohorts might be a real business trend.
The mistakes people make
A major red flag is jumping straight to a specific technical cause, like saying, "I'd check the git log for the reporting service." This is premature without validating the drop's existence. Another is blaming other teams, such as, "The data engineering pipeline must be broken," which is uncollaborative and ignores the prompt. Focusing only on technical signals (logs, traces) while ignoring business context (a competitor's launch, end of a promotion) shows a lack of business acumen. Finally, a purely technical plan with no mention of stakeholder communication is a failure for a senior role.
What usually comes next
Expect questions like: "How would you prevent this from happening again?" (Answer: data quality monitoring, anomaly detection on key metrics, schema contract testing). Or, "What if the drop was only 2%? How does your process change?" (Answer: lower urgency, less of a war room, but a similar systematic process). You may also be asked, "You find the bug. How do you handle backfilling the data and communicating the correction to leadership?"
A concrete example
A 20% drop in 'Daily Active Users' was reported. The pipeline was green. First, we checked raw server access logs and saw no drop in unique IP addresses, suggesting user activity was still there. Tracing the data, we found the ETL job that aggregates 'active user' events. A recent mobile app release had changed an event name from user_session_start to user_session_begun. The ETL filter was still looking for the old name, causing it to drop all events from users on the new app version. The fix was to update the ETL filter to include both event names and to backfill the aggregated data for the affected days.
Interview question
What is the most critical initial step when investigating a sudden 20% drop in a key revenue metric reported by a dashboard?
- a.Convene a war room with key stakeholders to brainstorm potential business impacts.
- b.Validate the reported drop by cross-referencing the dashboard's data with an independent, raw data source.Correct
- c.Examine the logs and metrics of the ETL/ELT pipelines populating the revenue dashboard.
- d.Immediately review recent code deployments to the reporting service for potential bugs.
Why? this is the answer
The card emphasizes that after initial communication, the second and critical step is to "validate the drop" by cross-referencing with an orthogonal source, as trusting the dashboard immediately is a red flag. Jumping to technical causes like code changes or ETL logs, or solely brainstorming business impacts without data validation, can lead to misdiagnosis or wasted effort.
Just read this? Test yourself on what you have been reading.
Read the original → newrelic.com
- #data integrity
- #incident response
- #analytics
- #systems thinking
- #leadership
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles