Observability
124 bites tagged Observability — interview questions with model answers, and 60-second explainers.
How do you debug a data quality issue in a complex pipeline?
Tests systematic debugging in distributed systems. A great answer prioritizes containment, traces data lineage from report to source, and proposes specific observability tools.
Explain data lineage and how you would implement it
This tests your ability to design for data observability. Define lineage (origin, transformation, movement), then propose a solution using metadata extraction (OpenLineage) and a central graph store/UI (Marquez) to trace data from microservices to analytics.
How would you measure P95 latency by geographic region?
Tests your ability to design a practical metrics pipeline, considering instrumentation, data types (metrics vs. logs), and aggregation. Instrument the API with a histogram metric and a `region` label, then query using `histogram_quantile`.
Why use p50/p95 over mean for API response times?
Tests your grasp of statistical distributions for UX metrics. A good answer explains how outliers skew the mean, while percentiles (p50, p95) better represent typical and worst-case user experiences.
How would you debug a critical data quality issue in a pipeline?
Tests your systematic debugging of a distributed system under pressure. A great answer contains the impact, traces data lineage backward from the report to the source, and then proposes preventative tooling.
Service Maps: A Live Blueprint of Your Architecture
A service map is a live blueprint of your distributed system, generated from telemetry data. It visualizes service dependencies and health, letting you instantly see an incident's blast radius instead of hunting through Slack for tribal knowledge.
Critical Path Analysis for Performance Tuning
Critical path analysis finds the slowest chain of operations in a request, showing where to optimize for impact. Use it in distributed tracing to see which service call is the bottleneck. Optimizing off-path components is wasted effort.
Loki: The Log System That Indexes Labels, Not Text
Loki is a log system that indexes only metadata labels, not the full log content. This makes it cheaper and simpler to run than full-text indexing systems, storing compressed logs in object storage.
Telemetry Correlation: Connecting Dots in Distributed Systems
Telemetry correlation links logs, metrics, and traces from a single user request into one coherent story. It works by propagating a shared context, like a unique ID, across all services involved.
Grafana: Your Single Pane of Glass for Observability
Grafana is the universal dashboard for your system's health, visualizing metrics, logs, and traces from different sources in one place. Use it to monitor application performance and infrastructure health.
Log Aggregation: Centralize Your System's Story
Log aggregation funnels scattered logs from many servers into one central, searchable system. It's crucial for debugging distributed systems where one request touches many services.
Prometheus Exemplars: Link Your Metrics to Traces
Exemplars are like footnotes for your metrics, linking a data point like a latency spike directly to a specific trace ID. This lets you jump from a 'what' on a dashboard to the 'why' in your tracing system.
Telemetry Processors: The Middle of the OTel Pipeline
A Telemetry Processor is a configurable stage in an OpenTelemetry Collector pipeline, sitting between data reception and export. You configure them in config.yaml to act on telemetry data.
Context Propagation: Stitching Microservices Together
Context propagation stitches a user request's journey across microservices by passing a shared ID. It's essential for distributed tracing, letting you see one request flow through many APIs.
OpenTelemetry Resource: The 'Who' of Your Telemetry
An OpenTelemetry Resource is the return address for your telemetry, describing the service instance that produced it. It attaches stable context like service name and version to every signal, enabling filtering.
OpenTelemetry Protocol (OTLP): The Universal Adapter for Telemetry
OTLP is the universal adapter for observability, providing a single vendor-neutral protocol for traces, metrics, and logs. Use it to send data from an OpenTelemetry SDK to a collector.
OpenTelemetry Semantic Conventions: A Shared Vocabulary
Think of Semantic Conventions as a universal dictionary for telemetry. They standardize attribute names like `http.method` across all services, letting you correlate data from different languages, libraries, and platforms seamlessly.
OpenTelemetry Collector: The Swiss Army Knife for Telemetry Data
The OpenTelemetry Collector is a universal adapter for your telemetry data. It receives data in one format, processes it, and exports it to multiple backends, letting you centralize configuration and avoid vendor lock-in.
OpenTelemetry SDK: The Engine for Your Telemetry
The OpenTelemetry SDK is the engine that processes and exports your telemetry data. It implements the OTel API, letting you configure how traces and metrics are sampled, batched, and sent to a backend.
OpenTelemetry API: The Stable Interface for Your Code
The OpenTelemetry API provides stable interfaces for your code to generate telemetry. This lets you instrument your application once, while the SDK implementation handles the actual data processing and export, which can be swapped out later.
Error Budget Burndown: Your Service's Reliability Fuel Gauge
An error budget burndown chart is a fuel gauge for your service's reliability. It shows how much 'allowable downtime' you've used, helping you decide when to ship features versus fix bugs. The footgun is ignoring a steep burn rate until it's too late.
Sampling: Tracing Everything Without Storing Everything
Sampling makes high-volume observability affordable by deciding which traces to keep and which to discard. It's essential in distributed systems where capturing every request is too costly.
Pull vs. Push: How Your Metrics Get to the Collector
Push vs. Pull metrics is about who starts the conversation. In a push model (like OTLP), the app sends metrics to a collector. In a pull model (like Prometheus), the collector scrapes metrics from the app. The footgun: pull can miss short-lived jobs.
The Four Golden Signals of Service Monitoring
The Four Golden Signals provide a simple framework for service health: Latency, Traffic, Errors, and Saturation. Use them for top-level dashboards and alerts. The biggest footgun is ignoring saturation, the leading indicator of future latency issues.
Get Observability bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.