Observability
124 bites tagged Observability — interview questions with model answers, and 60-second explainers.
Diagnosing a slow FastAPI endpoint under load
Add timing and tracing to isolate the slow span, watch CPU vs wait time and event-loop lag, then use profilers like py-spy or cProfile and DB EXPLAIN. Systematic performance diagnosis.
Propagating a correlation ID without parameter passing
Middleware reads or generates the header, stores it in a contextvars.ContextVar, service code reads it anywhere, and logging filters inject it. Ambient request-scoped context in async code.
Detecting and diagnosing event loop lag
Measure delay between scheduled and actual timer fire, expose it as a metric, find synchronous CPU-bound code. production diagnosis of a blocked single thread.
OpenTelemetry agent and gateway architecture
Agents run per-node for local collection and host enrichment; gateways are central, horizontally scaled pools for batching, tail sampling, and routing. Whether you can architect a scalable telemetry pipeline.
Designing shallow vs deep health checks
Shallow checks confirm the process is alive; deep checks verify dependencies; use shallow for liveness/load-balancer routing and deep sparingly to avoid… Whether you understand health-check semantics and failure amplification.
Conducting a Production Readiness Review
Assess monitoring and alerting, capacity and load testing, failure modes and dependencies, on-call and runbooks, and rollback or release safety. Whether you can systematically gate a launch on reliability.
What telemetry must a new microservice ship with?
Metrics for trends like p99 latency and error rate, structured logs for per-request detail, traces for cross-service causality. Practical observability literacy. Listing only host CPU dashboards and no request-level signals.
Canary a shared downstream microservice
Route a slice of traffic to the canary via mesh rules, propagate context, and use distributed tracing to measure impact on upstream callers across the full path. Mesh-based canary isolation.
Systematically reduce noisy alert toil
Inventory alerts, measure frequency, actionability, and time cost, then prioritize by volume times effort. Data-driven toil reduction. muting alerts blindly or automating before measuring which ones actually hurt.
Resolving post-mortem disagreement with data
Anchor the debate in the timeline, deploy events, traces, and metrics; correlate cause and onset; allow multiple contributing factors. Fact-based facilitation. resolving by seniority or opinion instead of evidence.
Architecting a single source of truth for incidents
One curated view overlaying golden metrics, deploy/change events, and active alerts on a shared timeline, fed by standardized telemetry. Consolidating incident signal. a wall of vanity panels nobody can read under pressure.
Technical investments to reduce MTTR
Cut detection, diagnosis, and recovery time via observability, runbooks/automation, fast rollback, and resilient architecture. Systemic MTTR reduction. only buying more alerting without addressing diagnosis or recovery speed.
Catching rare intermittent failures
Add high-cardinality traces and structured logs with exemplars, use tail-based sampling to keep failing traces, and alert on SLO burn rather than coarse averages. improving observability for rare events aggregates hide.
Correlating logs, metrics, and traces
Propagate a trace and span ID through context, stamp it on logs and metric exemplars, then pivot metric to trace to logs. knowing the trace ID is the glue across signals. correlating only by timestamp.
When to add custom OpenTelemetry instrumentation
Wrap domain logic in custom spans, attach business attributes like tenant or plan, answer revenue and per-customer questions. knowing auto-instrumentation only sees framework boundaries, not business meaning.
Redesigning a high-cardinality request metric
Cardinality is unique label combinations; user_id and raw path are unbounded; redesign by dropping user_id and templating the path. Diagnosing and fixing label explosion.
OpenTelemetry API, SDK, and Collector
API defines vendor-neutral instrumentation, SDK implements and exports it, Collector receives, processes, and routes telemetry to backends. Understanding the OTel pipeline.
Structured vs unstructured logging
Unstructured logs are free-text lines; structured logs are machine-readable key-value or JSON, enabling reliable parsing, filtering, and aggregation. Why log format matters at scale.
The three pillars of observability
Metrics show what and when via cheap aggregates, logs give detailed per-event context, traces show where across services. Knowing each telemetry type's strength.
Fixing watermelon SLOs that hide unhappy users
Green-outside, red-inside SLOs come from server-side measurement, coarse aggregation, wrong events, and lax thresholds; fix by measuring at the client and using meaningful SLIs. Whether SLIs truly track user experience.
Head-based vs tail-based trace sampling
Head-based decides at trace start cheaply but blindly; tail-based waits for the full trace to keep errors and slow requests. Tradeoffs in trace sampling. Claiming head-based can preferentially keep failed traces.
What is high-cardinality data in Prometheus?
Cardinality is the count of unique label combinations; each is a separate series; explosion blows up memory and query cost. Understanding time-series storage cost.
Black-box vs white-box monitoring for legacy apps
Black-box probes from outside, white-box reads internal state; start black-box for fast user-facing signal. Pragmatic monitoring strategy. Choosing white-box first on an uninstrumented system or ignoring black-box blind spots.
How does distributed tracing work?
Trace ID ties one request together, span IDs are individual operations, context propagates via headers. Whether you understand request correlation across services. Conflating traces with logs or omitting propagation entirely.
Get Observability bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.