What usability metrics would you instrument, and how do regressions drive priorities?

WHAT THIS TESTS: This question tests whether you can bridge user research and engineering execution. Interviewers want to see that you understand how to instrument task-level behavior, distinguish meaningful signals from noise, and translate a metric regression into a prioritized technical response. Senior engineers are expected to own outcomes, not just ship code.

A GOOD ANSWER COVERS: First, name the core metrics tied to task completion: success rate (can users finish the task), time on task (efficiency), error rate (slips and mistakes), and subjective satisfaction (SUS or CSAT). Second, mention efficiency metrics like optimal path deviation or backtracking frequency. Third, explain that instrumentation requires defining the task start and end states, logging interactions, and capturing post-task surveys. Fourth, describe how you handle a significant negative change: triage by the task's business criticality and the magnitude of the shift. A drop in checkout success rate is a stop-ship issue, while a slower admin configuration flow might warrant a backlog ticket. Fifth, state that quant metrics tell you what changed but rarely why, so you follow up with a small qualitative study of three to five users to diagnose root cause before pulling engineering resources. Sixth, show you understand cost: quantitative studies need roughly twenty users per design for tight confidence intervals, so you do not overreact to noisy data from small samples.

COMMON WRONG ANSWERS: Listing vanity metrics like page views or click counts instead of task-level usability metrics. Treating every regression as a P0 fire drill rather than triaging by impact. Confusing correlation with causation, such as assuming longer time on task always means worse usability when it might reflect deeper engagement. Ignoring the need for statistical significance and acting on tiny samples. Saying you would just hand the numbers to the design team without owning the engineering priority shift.

LIKELY FOLLOW-UPS: How do you balance quantitative metrics against qualitative insight in a tight timeline. How would you instrument these metrics in production versus a lab environment. What sample size and statistical test would you use to confirm a regression. Tell me about a time a metric dropped after your release and how you responded.

ONE CONCRETE EXAMPLE: Imagine an e-commerce checkout flow where task success rate drops from ninety-two percent to seventy-eight percent after a new payment UI ships. Your instrumentation captured step-level funnel drops and error messages. You treat this as a stop-ship priority, rolling back the UI or assigning a tiger team. In parallel, you run a five-user qualitative session and discover users misread a new billing address toggle. Engineering priority shifts from the next feature sprint to a hotfix, and you add guardrail metrics to prevent future checkout regressions.

Source: nngroup.com