Interview questions in Analytics & Metrics, page 16
Cohort analysis for an onboarding change
A cohort groups users by a shared start trait; compare pre and post Jan-1 signup cohorts on retention by age.
.png&w=1600&q=75)
Set up a cohort analysis for a new onboarding flow
This tests your ability to design a clean experiment to measure product impact. A great answer defines control/treatment cohorts by acquisition date (before/after Jan 1st), picks a specific metric like W1 retention, and compares them.
Cohort Analysis for a New Onboarding Flow
Tests applying analytics to measure impact. Define a cohort, then compare a pre-launch (Dec) vs. post-launch (Jan) acquisition cohort, tracking retention over time. A red flag is using aggregate metrics, which hide the true impact of the change.
Visualizing a correlation with a third variable
A scatter plot with a trend line shows the relationship; encode network type by color or facets to expose a lurking variable.
Visualize Correlation Between Load Time and Session Duration
Tests your ability to choose the right chart for correlation and layer in additional variables. A great answer starts with a scatter plot (load time vs. session duration), then uses color to represent the network type.
Visualizing Load Time vs. Session Duration with a Third Variable
Tests your ability to visualize correlation and add dimensions. A great answer suggests a scatter plot for the initial relationship, then uses color to segment by the categorical third variable (network type).
The multiple comparisons problem in A/B testing
Many tests at alpha 0.05 inflate the chance of a false positive; mitigate with Bonferroni or FDR control plus pre-registered metrics.
How do you handle the multiple comparisons problem in A/B testing?
Tests your grasp of statistical risk in experimentation. Explain how multiple tests inflate false positives, then describe mitigations like Bonferroni correction or limiting concurrent tests. A red flag is suggesting total test isolation, which is impractical.
What is the 'multiple comparisons problem' in A/B testing?
Tests your grasp of statistical pitfalls in large-scale A/B testing. Define the problem (inflated false positives), explain the business risk (wasted effort), and propose a mitigation like Bonferroni correction.
Implementing a consistent-assignment A/B test
Need an assignment service, exposure logging, and event tracking; ensure stickiness by hashing a stable user id; analyze conversion per variant.
Implement an A/B test for a new checkout flow
Tests your grasp of the full A/B testing lifecycle. A great answer outlines a feature flag system, consistent user bucketing via hashing a stable user ID, and an analytics query grouping by variant. A red flag is suggesting simple client-side randomization.
Implement an A/B test for a new checkout flow
This tests your ability to design a robust, stateful system for experimentation and data analysis. A great answer details user bucketing, consistent variant assignment across devices, and the SQL query structure for analysis.
SQL for a three-step onboarding funnel
Anchor the 30-day signup cohort, count distinct users reaching each later step in timestamp order; conversion is each step over the prior.
Design a User Onboarding Funnel Analysis System
This tests translating a business need into a data model and query. First, define the cohort. Then, use CTEs to find the first timestamp for each event per user. Finally, count users at each step.
Calculate a 3-step user onboarding funnel with SQL
Tests your ability to translate a business need into a robust data query. A great answer clarifies funnel logic (attribution, timing), defines the user cohort, finds each user's first event for each step, and then calculates conversion.
Architecting multi-touch attribution
Ingest touchpoints, resolve to one identity, order into paths, apply a model; last-touch is trivial, time-decay needs the full path.
Architect a Multi-Touch Attribution System
Tests your grasp of data pipeline trade-offs under real-world signal loss. A great answer outlines the pipeline (ingest, store, model), contrasts last-touch (simple state) vs.
Architect a Multi-Touch Attribution System
This tests your grasp of modern data challenges like signal loss. A good answer discusses data ingestion, identity resolution, and model trade-offs. A red flag is focusing only on the algorithm and ignoring the data pipeline's fragility.
Client-Side vs. Server-Side Event Tracking
This tests your grasp of data integrity trade-offs. A great answer advocates for server-side tracking for critical events due to its reliability against ad blockers, using client-side only for supplementary UI events. A red flag is treating them as equal.
Client-side vs. Server-side Event Tracking: When and Why?
This tests your grasp of data reliability and security trade-offs. A good answer defines both, contrasts reliability (ad blockers) vs. implementation ease, and uses a critical event like "Payment Processed" to justify server-side's accuracy.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles