Experiment Knowledge Base: Don't Repeat Failed Tests

WHY IT EXISTS: As an organization scales its A/B testing, institutional knowledge gets fragmented across teams and documents, or lost entirely when people leave. Without a central system, teams waste time and resources re-running experiments or building on flawed assumptions. A knowledge base exists to create a permanent, shared memory of experimental learnings to accelerate innovation.

THE MENTAL MODEL: Think of it as a shared lab notebook for your entire company. A single scientist uses a notebook to avoid repeating work and to build on past discoveries. An Experiment Knowledge Base does the same for an organization, ensuring that every test, whether it succeeds or fails, contributes to a collective intelligence.

HOW IT WORKS: It is a structured, searchable repository where every experiment is documented. A typical entry includes the experiment's name, its owner, the core hypothesis, the specific changes made, the target audience, and the success metrics. Crucially, it also includes the statistical results and a human-written summary of the key learnings and decisions made. This allows anyone to find past tests related to a feature or metric and understand the outcome.

WHEN TO USE IT: A knowledge base becomes critical when your organization runs more experiments than a single person can track. Companies that run thousands of tests a year, like Microsoft or Google, rely on these systems to manage complexity. Use it before planning any new experiment to see what has been tried before, what was learned, and why certain product decisions were made.

WHEN NOT TO USE IT: For a tiny team running only a handful of experiments, the overhead of a formal system may not be worth it; a simple spreadsheet or shared document might suffice. The need for a dedicated knowledge base grows with the volume and complexity of your experimentation program.

ONE CANONICAL EXAMPLE: A product manager wants to test a new checkout flow. Before designing it, she searches the knowledge base for “checkout” and “conversion.” She finds three past experiments. One failed because it added too much friction. Another showed a small lift but was never shipped due to a technical dependency. A third revealed that customers on mobile responded poorly to a similar change. This context saves her weeks of work and helps her design a much smarter fourth experiment.