Measuring post-incident review effectiveness

WHAT THIS TESTS Whether you can measure a soft process rigorously while avoiding metrics that create perverse incentives like underreporting.

A GOOD ANSWER COVERS Start from the outcome the process exists to produce: fewer and less severe incidents over time. The strongest effectiveness signal is the repeat-incident rate, how often the same root cause or class recurs, ideally trending down, alongside an improving MTTR trend. For process health, define SLIs such as action-item completion rate and aging distribution, time from incident resolution to published post-mortem, and review coverage, the share of qualifying incidents that actually get a review. You can also track action-item lead time and the ratio of systemic to superficial fixes. Crucially, choose metrics that do not punish honesty: never reward fewer incidents reported, since that drives underreporting, and never blame individuals via the metrics.

COMMON WRONG ANSWERS Vanity counts like number of post-mortems written. Metrics that incentivize hiding or downgrading incidents. Measuring activity instead of outcomes.

LIKELY FOLLOW-UPS How do you attribute a repeat incident to a missed action item? What target do you set for time-to-publish? How do you prevent metric gaming?

ONE CONCRETE EXAMPLE You instrument the process with four signals: repeat-incident rate by root-cause category (down quarter over quarter is the goal), action-item completion within 30 days, median time from resolution to published post-mortem (target a few business days while context is fresh), and review coverage for SEV1 and SEV2 events. You deliberately avoid rewarding a lower raw incident count, because that would tempt teams to stop declaring incidents, which destroys the very signal the review process depends on.