Architecting a single source of truth for incidents

WHAT THIS TESTS Whether you can design observability for decision-making during chaos, not just data collection. The goal is to align responders on one picture.

A GOOD ANSWER COVERS A single source of truth is a curated, opinionated view, not a dump of every metric. Put the golden signals, latency, traffic, errors, and saturation, for the critical services on a shared, synchronized time axis. Overlay a change feed: deploys, config flags, feature toggles, and infra changes, since correlating symptoms with changes is the fastest path to cause. Overlay active alerts and the incident timeline too. Underneath, standardize telemetry: consistent labels, a service catalog, and a metrics naming convention so data from different teams joins cleanly instead of conflicting. Make this dashboard the canonical artifact named in the incident runbook, with the incident channel linking to it, so debate happens against shared facts.

COMMON WRONG ANSWERS A 50-panel wall nobody can parse under stress. Letting each team bring its own dashboard, reproducing the conflict. Omitting deploy and change events, the highest-signal overlay.

LIKELY FOLLOW-UPS How do you standardize labels across teams? How do you keep the view fast when systems are degraded? Who owns and curates it?

ONE CONCRETE EXAMPLE Three teams argue: networking blames a deploy, the app team blames the database, ops blames load. The canonical dashboard overlays error rate, the deploy feed, and saturation on one timeline. It shows errors began exactly at a config push, not at any traffic change or DB event, ending the debate in seconds. Because every service emits standardized labels, the panels line up, and the incident channel header links straight to that view so every joiner sees the same truth.