Metrics
400 bites tagged Metrics — interview questions with model answers, and 60-second explainers.
CFD Testing band widens: what does it indicate and what experiments?
This tests flow metric literacy. A widening Testing band means arrivals exceed departures; propose experiments like smaller batches, automation, or dev-test swarming, then measure cycle time. Red flag: blaming testers or demanding headcount without data.
Lead Time vs Cycle Time in Kanban and measuring Cycle Time
Tests whether you distinguish customer wait time from active work. Strong answer: Lead Time is request-to-delivery with queues; Cycle Time is active start-to-finish measured from In Progress to Done. Red flag: treating them as synonyms or ignoring wait states.
How would you measure a launched feature's success and impact?
This tests if you link code to business outcomes via agile metrics. A strong answer covers value, quality, satisfaction; names metrics like velocity or cycle time; and uses reports to track progress. Red flag: defining success purely by uptime or bug counts.
How would you implement an A/B test for a redesigned dashboard?
Tests experiment rigor: randomize stable user buckets, pick one primary metric and guardrails like latency, pre-calculate sample size and runtime with a minimum detectable effect, and validate via backend logging.
Define and calculate Weekly Active Users (WAU) for Slack
Tests translating a business metric to a technical spec. Define 'active' by key actions (sending messages, not just opening), then `COUNT(DISTINCT user_id)` on an events table, filtering out bots and background syncs. A red flag is a generic definition.
When is an A/B test not feasible, and what is DiD?
This tests your grasp of causal inference when randomization isn't possible. Explain a scenario like a state-level launch, introduce Difference-in-Differences (DiD), and state its core parallel trends assumption.
Determine Sample Size for a 2% Lift A/B Test
This tests your grasp of statistical power and the business trade-offs in experimentation. A great answer defines baseline conversion rate, minimum detectable effect (MDE), and statistical power. A red flag is ignoring the business context of MDE.
Calculate Monthly Recurring Revenue (MRR) with SQL
This tests your ability to translate a business metric into a robust SQL query, handling time-series logic. A good answer filters for active subscriptions, sums the price, and correctly amortizes annual plans. A red flag is using incorrect date filtering.
What is a p-value, and what does 0.03 practically mean?
This tests your ability to translate stats into business decisions. A great answer defines p-value, compares 0.03 to the standard 0.05 threshold to reject the null hypothesis, and recommends shipping.
Propose a multi-touch attribution model and its data pipeline
This tests your ability to choose a practical data model under real-world constraints. Propose a time-decay or position-based model, then describe the data pipeline: event collection, identity resolution, and aggregation. A red flag is ignoring signal loss.
How would you find leading indicators for long-term churn?
Tests your ability to connect a lagging business KPI to leading product metrics. A good answer defines churned/retained cohorts, analyzes first 30-day engagement differences (e.g., feature adoption), and validates findings. A red flag is jumping to ML models.
Calculate Daily Active Users (DAU) with SQL
This tests your ability to translate a business metric into a precise technical definition and query. A good answer defines "active," specifies the event data needed (user_id, timestamp, event_name), and uses COUNT(DISTINCT user_id).
Client-Side vs. Server-Side Event Tracking
This tests your grasp of data integrity trade-offs. A great answer advocates for server-side tracking for critical events due to its reliability against ad blockers, using client-side only for supplementary UI events. A red flag is treating them as equal.
Visualize Correlation Between Load Time and Session Duration
Tests your ability to choose the right chart for correlation and layer in additional variables. A great answer starts with a scatter plot (load time vs. session duration), then uses color to represent the network type.
Set up a cohort analysis for a new onboarding flow
This tests your ability to design a clean experiment to measure product impact. A great answer defines control/treatment cohorts by acquisition date (before/after Jan 1st), picks a specific metric like W1 retention, and compares them.
How to visualize a complex, multi-stage customer funnel?
Tests your ability to choose the right visualization for non-linear user flows. Propose a Sankey or Alluvial diagram to show flow volume, drop-off, and re-entry. A red flag is suggesting multiple simple charts that fail to show the paths between stages.
Apply the AARRR framework to B2B SaaS vs. B2C mobile games
Tests translating the AARRR framework into concrete analytics for different business models. Define AARRR, then contrast B2B SaaS (account-level activation) with B2C games (user-level engagement). Red flag: using identical metric definitions for both contexts.
What is the difference between a metric and a KPI?
This tests your ability to connect technical measures to business outcomes. Define metrics as operational data and KPIs as the subset tied to critical goals.
A key metric dropped 20%. How would you investigate?
This tests systematic diagnosis of critical issues. A great answer segments the drop (by region, platform), then traces data upstream from the dashboard to the source, correlating with technical metrics. A red flag is jumping to code before scoping the impact.
Propose a technical architecture for a centralized Metrics Layer.
This tests your grasp of data governance and semantic layers. A great answer outlines a system with a central definition store (e.g., YAML in Git), a query engine, and an API, ensuring all teams get consistent metric results.
How would you build a weekly active user dashboard?
This tests translating a business need into a technical spec. First, clarify the definition of "active." Then, identify the event data needed (user_id, timestamp). Finally, outline the ETL and BI tool steps. A red flag is not clarifying the metric first.
Design an experiment to isolate long-term impact from novelty effect
Tests if you can design experiments for long-term impact, not just short-term lift. A good answer involves a long-running test, segmenting users by tenure, and modeling the effect over time to find its stable asymptote.
Explain cohort analysis and write a retention query
Tests your grasp of user behavior metrics. First, define a cohort (e.g., May signups). Then, explain why it's better than aggregate retention. Finally, outline a query grouping users by signup week and checking for activity. A red flag is a flawed query.
Track an 'Export to CSV' button's usage
This tests modeling a feature's full lifecycle (intent, success, failure) in analytics. Propose one custom event like `export_interaction` with a `status` parameter ('success', 'failure') fired on completion, not just on click.
Get Metrics bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.