Diagnosing a slow FastAPI endpoint under load

WHAT THIS TESTS Whether you debug performance scientifically rather than guessing, and whether you understand how blocking work harms an async server differently from a thread-per-request one.

A GOOD ANSWER COVERS Start by measuring, not editing. Add structured timing around suspect spans or use OpenTelemetry to trace a request end to end and see where wall-clock time concentrates: the database, an external API, serialization, or your own code. Collect latency percentiles and concurrency from a metrics system so you reason about behavior under load, not a single call. To classify the bottleneck, watch system signals: if a CPU core sits near 100 percent and the asyncio event loop shows scheduling lag, the work is CPU-bound, often heavy serialization, crypto, or a sync function blocking the loop. If CPU is mostly idle yet latency is high while many requests sit waiting, it is I/O-bound, typically slow queries, an unindexed table, a saturated connection pool, or a slow upstream. Tools include py-spy, which samples a running process with low overhead and is safe in production, cProfile or Pyinstrument for local deep dives, EXPLAIN ANALYZE for SQL, and event-loop monitors that flag long-running coroutines.

COMMON WRONG ANSWERS Jumping straight to caching or adding workers before measuring. Equating high latency with high CPU, when the server may be idle and waiting on I/O. Running cProfile in production where its overhead skews results. Ignoring that one blocking sync call stalls every coroutine on that worker.

LIKELY FOLLOW-UPS How does a blocking call differ in impact between sync and async endpoints? How do you offload CPU-bound work? What does event-loop lag indicate?

ONE CONCRETE EXAMPLE Latency is high but CPU is 10 percent, so it is I/O-bound; a trace shows 400 ms inside one query. EXPLAIN ANALYZE reveals a sequential scan; adding an index drops it to 5 ms. Separately, a 100 percent CPU spike traced by py-spy points to synchronous bcrypt hashing on the event loop, which you move to run_in_executor.