Profiling Rust with Linux perf

WHY IT EXISTS: Developers optimizing Rust often guess wrong about where CPU time goes. Adding print statements or instrumentation libraries changes timing and hides real behavior. perf solves this by using Linux kernel facilities to observe a running process from the outside with minimal overhead, giving an unbiased view of actual execution.

THE MENTAL MODEL: Think of perf as a strobe light pointed at your CPU. It flashes thousands of times per second and takes a snapshot of the instruction pointer and call stack each time. After enough flashes, the areas that appear brightest are where your program lingers. It is statistical, not exact, but because the samples are triggered by hardware performance counters, the overhead is low enough to run against production traffic.

HOW IT WORKS: You run perf record against your Rust binary. The kernel programs a hardware counter, usually CPU cycles, to trigger an interrupt after a set number of events. When the interrupt fires, the kernel captures the current stack. For Rust, the binary should include debug symbols so addresses can be mapped back to function names, though the symbols do not need to remain in the shipped artifact if you keep them separately. Because Rust omits frame pointers by default to free up a register, perf often cannot walk the stack with its default frame-pointer mode. You either recompile with RUSTFLAGS set to force frame pointers or tell perf to use DWARF debug info for stack unwinding via the call-graph dwarf option, which is more accurate but heavier. After recording, perf report or perf script aggregates samples and shows inclusive and exclusive cost per function.

WHEN TO USE IT: Use perf when you have a CPU-bound Rust service on Linux that is consuming more cores than expected, when you need to compare before-and-after profiles of an optimization, or when you suspect cache misses or branch mispredictions and want to sample by specific hardware counters rather than wall time.

WHEN NOT TO USE IT: Do not rely on perf for very short-lived programs because you will not gather enough samples for statistical significance. It is also less helpful for off-CPU analysis such as tracking async task scheduling delays or blocking IO waits unless you explicitly switch to scheduler events. On non-Linux systems, perf is not available, and tools like Instruments on macOS or VTune are the alternatives. Finally, if you need exact call counts rather than statistical sampling, perf will not give you precise invocation numbers.

ONE CANONICAL EXAMPLE: You have a Rust HTTP server handling ten thousand requests per second and latency spikes. You build a release binary with debug symbols enabled in Cargo.toml, then run perf record with call-graph dwarf against your release server. After loading test traffic for sixty seconds, you stop recording and run perf report. The report shows thirty percent of samples landing inside a specific deserialization function. You inspect the call graph and see it is called from a middleware loop. You refactor the loop to reuse a buffer, rebuild, re-profile, and confirm the function drops to five percent of samples.