Criterion: Statistical Benchmarking for Rust

Why it exists

Simple timers give noisy results. Was your code faster, or was the OS just less busy for a moment? Criterion was built to answer this question with statistical confidence, moving beyond the limitations of basic benchmark runners like Rust's built-in libtest bencher.

The mental model

Think of Criterion as a science lab for your functions. A simple benchmark is like timing a sprint once with a phone. Criterion is like a full athletic combine: it warms up the runner, takes many measurements under controlled conditions, discards flukes, and produces a detailed report on their true average speed and consistency.

How it works

When you run cargo bench, Criterion takes over. For each benchmark, it performs a warm-up phase, then runs the code in a loop to collect thousands of timing samples. It analyzes this data to find the mean time and confidence intervals. Crucially, it saves these results. On subsequent runs, it compares the new statistics to the old ones and reports any statistically significant changes, flagging performance regressions automatically. It visualizes all this in detailed HTML reports.

When to use it

Use Criterion when performance is critical and "I think it's faster" isn't good enough. It's perfect for proving an optimization on a hot-path function, comparing the performance of two different algorithms side-by-side, or integrating into CI to automatically catch performance regressions before they merge.

When not to use it

Criterion excels at microbenchmarking—measuring small, self-contained functions. It is not a tool for system-wide, end-to-end profiling. If your function's performance is dominated by network calls, disk I/O, or database queries, a dedicated profiler will give you a more accurate picture of the real bottlenecks.

One canonical example

A classic use is comparing two implementations. You can create a benchmark group with two functions: one for a recursive Fibonacci solver and one for an iterative one. Criterion will run both, analyze their performance distributions, and generate a report that not only says which is faster but by how much, complete with graphs and statistical confidence levels.