Interview questions in Analytics & Metrics, page 19
Determine Sample Size for a 2% Lift A/B Test
This tests your grasp of statistical power and the business trade-offs in experimentation. A great answer defines baseline conversion rate, minimum detectable effect (MDE), and statistical power. A red flag is ignoring the business context of MDE.
Calculate Sample Size for a 2% A/B Test Lift
This tests if you connect statistical inputs to business goals. A good answer defines baseline rate, minimum detectable effect (MDE), and power, then explains MDE as a cost/benefit trade-off.
Use Difference-in-Differences without an A/B test
Give a scenario like a region-wide launch, apply Difference-in-Differences comparing treated vs control over time, and state the parallel-trends assumption.
When is an A/B test not feasible, and what is DiD?
This tests your grasp of causal inference when randomization isn't possible. Explain a scenario like a state-level launch, introduce Difference-in-Differences (DiD), and state its core parallel trends assumption.
When is A/B testing not feasible, and what is an alternative?
Tests your grasp of causal inference when randomization isn't possible. A great answer names a scenario (like a regional launch), proposes Difference-in-Differences (DiD), and explains its core 'parallel trends' assumption.
Define and calculate Weekly Active Users
Define a meaningful active action, count distinct users over a rolling 7-day window, and exclude bots and background syncs.
Define and calculate Weekly Active Users (WAU) for Slack
Tests translating a business metric to a technical spec. Define 'active' by key actions (sending messages, not just opening), then COUNT(DISTINCT user_id) on an events table, filtering out bots and background syncs. A red flag is a generic definition.
How would you define and calculate Weekly Active Users (WAU)?
This tests your product sense and technical precision in defining a core business metric. A great answer defines 'active' with specific user actions, outlines the SQL/event-based calculation, and discusses pitfalls like bots and background events.
How do you shift analytics from growth to profitability?
This tests your ability to translate business strategy into technical changes. A great answer connects profitability drivers to specific updates in event taxonomy, data models, and dashboards. A red flag is ignoring core financial metrics like LTV and CAC.
Define idempotency in data processing and give an example
Tests your grasp of distributed systems reliability. Define idempotency (N>1 runs = 1 run), explain its role in fault-tolerant retries, and provide a concrete example using transaction IDs. A red flag is confusing it with immutability.
How would you visualize three years of monthly revenue?
This tests your grasp of time-series visualization and data integrity. A strong answer picks a line chart, insists on a zero-based Y-axis and clear labels, and adds context like seasonality.
Visualize two continuous and one categorical variable?
Tests your ability to map data to visual encodings. A great answer starts with a scatter plot, then adds the categorical data using color, shape, or faceting, explaining the tradeoffs. A red flag is suggesting a 3D chart, which is difficult to read.
SARIMA vs. LightGBM for Forecasting with External Variables
Tests your grasp of practical trade-offs in model selection. A strong answer contrasts SARIMA's interpretability with LightGBM's power for handling many non-linear variables, covering performance and implementation costs.
A key metric dropped 15%. How do you investigate?
This tests systematic debugging of business metrics. A great answer first validates the data itself, then checks for recent changes (deploys, features), and finally segments the drop to isolate the cause. A red flag is immediately assuming a product bug.
Design a Privacy-Compliant Analytics Architecture
This tests your ability to balance data utility with strict privacy controls. A great answer outlines a central governance layer, dynamic masking, and purpose-based access tied to auditable logs.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles