Should we build or buy an A/B testing platform?
This tests build-vs-buy judgment for experimentation infrastructure. Strong answers cover build for warehouse metrics and cache control; buy for proven stats and front-end speed. Red flag: answering with cost alone or assuming in-house is always superior.
What's really being asked
This question probes whether you can move beyond ideological build-versus-buy debates and instead map specific technical capabilities to business outcomes. The interviewer wants to see that you understand experimentation infrastructure as a trust and velocity problem, not merely a procurement decision.
The full answer
A strong response first anchors the build case on integration depth and data ownership. You should mention that an in-house platform can tie directly into your existing data warehouse, letting you test against custom aggregate metrics or leading indicators rather than being limited to standard binomial events. It also allows tight coupling with your deployment pipeline and feature flags, which speeds up complex back-end experiments and avoids the cache-invalidation and page-flicker issues that plague third-party front-end injection. Then you should pivot to the buy case, emphasizing time-to-first-test measured in hours rather than months, a proven statistical engine that builds leadership trust, and the ability for non-engineers to launch front-end tests without creating an engineering bottleneck. You should also note that buying introduces risks around data privacy because user events are shipped to an external vendor, and that off-the-shelf metric models are often rigid.
The mistakes people make
The most common red flag is answering with cost alone or assuming in-house is always better because you can customize it. Another weak pattern is ignoring statistics entirely; a senior candidate should acknowledge that building a trustworthy stats engine requires expertise and that buggy in-house systems destroy experiment credibility. Similarly, praising third-party tools without mentioning warehouse-native limitations or page-flicker latency signals shallow evaluation.
What usually comes next
Expect the interviewer to ask how you would mitigate the downsides of your chosen path. If you advocate buying, they may ask how you prevent test collision and maintain strategic guardrails when many teams can launch experiments. If you lean toward building, they may ask for a rough timeline or how you would validate the statistical engine before shipping it to production.
A concrete example
Suppose you run a high-traffic e-commerce site with heavy edge caching and a Snowflake warehouse containing a custom lifetime-value model. Building in-house lets you assign variants at the edge without cache-busting and compute experiment results directly against your LTV model. Conversely, if you are an early-stage SaaS startup with no data warehouse and a small engineering team, buying lets you start running front-end copy tests tomorrow while your engineers focus on the core product.
Interview question
Which pair of constraints most strongly favors building an in-house A/B testing platform over purchasing a third-party solution?
- a.Heavy reliance on edge caching and need to experiment against custom warehouse-native metricsCorrect
- b.Desire for fastest time-to-first-test and limited in-house data warehouse infrastructure
- c.Small engineering team and urgent need to run front-end copy tests without engineering bottlenecks
- d.Need for proven statistical credibility and requirement that non-engineers launch front-end tests
Why? this is the answer
Building is optimal when you need tight control over edge caching and must compute experiment results against custom warehouse metrics like LTV that off-the-shelf tools cannot support. Option D is tempting because statistical credibility is critical, but the card warns that building a trustworthy stats engine requires deep expertise and buggy in-house systems destroy credibility, making buying preferable there.
Just read this? Test yourself on what you have been reading.
Read the original → growthbook.io
- #experimentation
- #growth engineering
- #infrastructure
- #a/b testing
- #product engineering
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on experimentation — each one lists the topics its interview covers.
See open roles