Should we build or buy an A/B testing platform?

WHAT THIS TESTS: This question probes whether you can move beyond ideological build-versus-buy debates and instead map specific technical capabilities to business outcomes. The interviewer wants to see that you understand experimentation infrastructure as a trust and velocity problem, not merely a procurement decision.

A GOOD ANSWER COVERS: A strong response first anchors the build case on integration depth and data ownership. You should mention that an in-house platform can tie directly into your existing data warehouse, letting you test against custom aggregate metrics or leading indicators rather than being limited to standard binomial events. It also allows tight coupling with your deployment pipeline and feature flags, which speeds up complex back-end experiments and avoids the cache-invalidation and page-flicker issues that plague third-party front-end injection. Then you should pivot to the buy case, emphasizing time-to-first-test measured in hours rather than months, a proven statistical engine that builds leadership trust, and the ability for non-engineers to launch front-end tests without creating an engineering bottleneck. You should also note that buying introduces risks around data privacy because user events are shipped to an external vendor, and that off-the-shelf metric models are often rigid.

COMMON WRONG ANSWERS: The most common red flag is answering with cost alone or assuming in-house is always better because you can customize it. Another weak pattern is ignoring statistics entirely; a senior candidate should acknowledge that building a trustworthy stats engine requires expertise and that buggy in-house systems destroy experiment credibility. Similarly, praising third-party tools without mentioning warehouse-native limitations or page-flicker latency signals shallow evaluation.

LIKELY FOLLOW-UPS: Expect the interviewer to ask how you would mitigate the downsides of your chosen path. If you advocate buying, they may ask how you prevent test collision and maintain strategic guardrails when many teams can launch experiments. If you lean toward building, they may ask for a rough timeline or how you would validate the statistical engine before shipping it to production.

ONE CONCRETE EXAMPLE: Suppose you run a high-traffic e-commerce site with heavy edge caching and a Snowflake warehouse containing a custom lifetime-value model. Building in-house lets you assign variants at the edge without cache-busting and compute experiment results directly against your LTV model. Conversely, if you are an early-stage SaaS startup with no data warehouse and a small engineering team, buying lets you start running front-end copy tests tomorrow while your engineers focus on the core product.

Source: Growthbook Blog, "A/B testing platforms: build vs buy" by Graham McNicoll, Nov 23, 2020.