Concepts in Product Management, page 19
Server-Side Experimentation: Testing Your Backend Logic
Server-side experimentation renders A/B test variations on the server before sending the page. Use it for testing deep backend logic like search algorithms or to avoid the visual 'flicker' of client-side tests. The footgun: it requires developer cycles.
Average Treatment Effect (ATE): Isolating the Impact of a Change
The Average Treatment Effect (ATE) isolates an intervention's true impact by comparing the average outcome of a treated group to a control group. It's used in A/B tests and policy evaluations. The footgun is assuming causation without true randomization.
Jobs to be Done: Sell the Hole, Not the Drill
Jobs to be Done (JTBD) says people don't buy a drill, they buy a hole. It focuses on the customer's underlying goal, not your product. This helps uncover new opportunities, but the footgun is defining the job too narrowly, limiting innovation to your current…
Metrics Layer: The Dictionary for Your Data
A metrics layer is the central dictionary for your company's numbers, defining what "Revenue" or "Active User" means once for everyone. It ensures teams and AI agents get consistent answers from a single source of truth, preventing conflicting reports.
The Novelty Effect: When New Isn't Always Better
The Novelty Effect is a temporary metric spike from a feature's newness, not its inherent value. It often appears in A/B tests for high-frequency products, inflating short-term metrics. The footgun is mistaking this initial excitement for a long-term win.
Design Sprint: From Idea to Prototype in 5 Days
A Design Sprint fast-forwards you to see customer reactions without building a real product. In one week, your team goes from a big question to a realistic prototype and user feedback.
Product Ecosystem Strategy: Competing Through Collaboration
A product ecosystem strategy builds a network of collaborating and competing partners around a core offering, making the whole more valuable than any single part. It's the model behind app stores and smart home devices.
What is an Experimentation Stats Engine?
A stats engine is the brain of an A/B testing platform, turning raw data into reliable 'which version won?' decisions. It powers tools that analyze feature rollouts, ensuring statistical rigor.
Sample Ratio Mismatch (SRM): When Your A/B Test Is Broken
Sample Ratio Mismatch (SRM) means your A/B test's traffic split is broken, violating random assignment. For example, a 50/50 split results in a statistically significant imbalance.
OKRs: Set Direction, Not Tasks
OKRs connect ambitious vision (Objectives) to measurable progress (Key Results). Used by companies to align quarterly efforts on high-level goals, not just a feature list.
Portfolio Balancing: Don't Bet Everything on One Project
Portfolio balancing treats your projects like an investment portfolio, diversifying bets. It's used to allocate engineers between new features, tech debt, and R&D. The footgun is only funding short-term wins, starving long-term health and innovation.
Growth Product Manager: Driving Metrics, Not Just Features
A Growth PM is a business optimizer for an existing product. They focus on moving a single metric like user activation or retention, often through rapid experimentation. This role is key in product-led companies where the product must sell itself.
Multi-Armed Bandit: The Explore vs. Exploit Trade-off
A multi-armed bandit algorithm balances exploring new options with exploiting the current winner, like a gambler trying slot machines to find the best payout.
Dual-Track Agile: Discovery Before Delivery
Dual-Track Agile runs two parallel streams: a Discovery track to quickly validate ideas and a Delivery track to build releasable software. It prevents waste by testing concepts with cheap prototypes before writing code.
Platform Business Model: Connecting, Not Owning
A platform business connects groups (like buyers and sellers) instead of making its own products. It's a marketplace, not a store. This model powers ride-sharing apps and app stores.
Growth Meeting Cadence: Focus on Learnings, Not Updates
Run your weekly growth meeting like a learning synthesizer, not a status report. Focus on extracting insights from experiments to drive future impact. The biggest mistake is wasting time on "what" you're doing; handle status updates asynchronously.
Isolating Impact with Difference-in-Differences (DiD)
Difference-in-Differences (DiD) isolates an intervention's true effect by comparing a treatment group's change over time to a control group's. This reveals if a new feature truly boosted engagement, not just rode a general upward trend.
Pretotyping: Build The Right It, Not Just It Right
Pretotyping tests if anyone wants your product before you build it, ensuring you build 'The Right It.' Use it to validate market demand with minimal resources, gathering real data. The footgun is confusing it with prototyping, which tests implementation.
Two-Sided Markets: Playing Matchmaker for Value
A two-sided market plays matchmaker, connecting two distinct groups like buyers and sellers. It creates value by enabling their interaction, growing stronger as more users from each side join.
The Independent Growth Team Model
An independent growth team is a startup-within-a-company, given autonomy to run experiments across the funnel. Use it to break silos and accelerate learning. The main footgun is isolation, creating a rogue unit whose wins are difficult to integrate.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles