Detecting and fixing metric hacking

WHAT THIS TESTS The interviewer is probing whether you understand Goodhart's law: when a measure becomes a target it ceases to be a good measure. They want technical instincts for detecting the divergence and a fix grounded in instrumentation and KR design, not blame.

A GOOD ANSWER COVERS Look for the proxy and the true goal moving in opposite directions: time-on-page up while task completion, return rate, NPS, or satisfaction fall. Hunt for the mechanism in the data: rising rage-clicks, longer time-to-complete a core task, increased back-navigation, pagination that splits content to inflate page views, or confirm-shaming dialogs. Then realign the metric design. Pair each KR with guardrail counter-metrics that must not regress, redefine the KR toward a truer outcome such as successful task completion rather than raw dwell time, and segment by cohort to reveal whether gains come from confused users rather than satisfied ones.

COMMON WRONG ANSWERS Defending the single metric because the KR is technically green. Accusing the team of cheating rather than recognizing the proxy invited the behavior. Adding more of the same kind of vanity metric. Proposing only a survey while leaving the gameable KR in place.

LIKELY FOLLOW-UPS How do you choose a guardrail metric? How would you instrument task success specifically? How do you prevent the next proxy from being gamed too?

ONE CONCRETE EXAMPLE You notice time-on-page rose 30 percent but task completion fell and support tickets climbed. Drilling in, the team had split an article across more pages and added interstitials, inflating dwell time while frustrating users. You propose redefining the KR to measured task completion, adding a guardrail that satisfaction and completion may not drop, and instrumenting rage-clicks so the gaming pattern is visible next time.