Observability
124 bites tagged Observability — interview questions with model answers, and 60-second explainers.
Structured vs unstructured logging: why it matters
Unstructured logs are free-text lines hard to parse; structured logs are machine-readable key-value or JSON records; structure enables reliable querying… understanding of log formats and their impact on observability at scale.
Explain the three pillars of observability
Metrics are cheap aggregated time series for detecting and alerting, logs are discrete event records for detail and context, traces follow one request across services to localize latency. practical command of metrics, logs, and traces.
Baggage in Distributed Tracing
Baggage is key-value context propagated alongside a trace across service boundaries, so downstream services can read values set upstream. It enables cross-cutting context like tenant id, but it travels in headers on every hop, so overuse adds latency and leak…
Auto-Instrumentation
Auto-instrumentation automatically adds telemetry to an application without manual code changes, by hooking into libraries, frameworks, or the runtime. It gives broad baseline observability fast, but produces generic spans that often need manual…
Metrics in Observability
Metrics are numeric measurements aggregated over time, like counters, gauges, and histograms. They are cheap to store and fast to query, making them ideal for dashboards and alerting, but their pre-aggregation discards per-event detail needed for deep…
Observability vs Monitoring
Monitoring watches predefined metrics and alerts on known failure modes you anticipated. Observability is the property of being able to ask new questions about a system's internal state from its outputs, letting you debug unknown failures you never predicted…
Viewing pod logs and durable log collection
Kubectl logs (with -c, --previous, -f) reads container stdout/stderr; because that storage is ephemeral, run a node-level logging agent as a DaemonSet shipping logs to a central store. Logging basics and durability.
Diagnosing database latency layer by layer
Split total time into pool-wait, query execution, and ORM-generated query patterns; use metrics at each layer. structured latency diagnosis.
Diagnosing slow auto-scaled PaaS workloads
Application metrics like request latency, throughput, and DB query time; infrastructure metrics like CPU, memory, and scaling lag. layered debugging under load. jumping to add instances without isolating the real bottleneck.
Difference between metrics and logs
Metrics are aggregated numeric time series good for trends and alerting; logs are discrete timestamped event records good for detailed root-cause analysis. observability fundamentals.
Centralized logging across microservices
Ship structured logs from every service into a central searchable store, then propagate a correlation ID through all hops to trace one request. aggregating and correlating logs. grepping per-host files with no shared ID.
Distributed tracing for serverless apps
Propagate a trace context through every hop, instrument with OpenTelemetry or X-Ray, capture spans including the database, and correlate traces with structured logs. end-to-end observability. per-service logs, no correlation.
Diagnosing Go memory leaks with pprof heap profiles
Expose net/http/pprof, grab /debug/pprof/heap, analyze inuse_space for live retention versus alloc_space for cumulative allocation; rising inuse over time points to a leak. production profiling with pprof.
Diagnosing latency with distributed tracing
Follow the trace ID across spans, compare per-span durations to find the slow hop, distinguish service time from network and queueing. using traces to localize a bottleneck.
What a service mesh solves
It adds traffic management, security via mTLS, and observability at the network layer through sidecars, beyond what plain Kubernetes offers. the purpose of a mesh.
Head-based vs tail-based trace sampling
Head decides up front (cheap, may miss rare errors); tail decides after the trace completes (catches errors and slow traces but needs buffering). trace sampling trade-offs. claiming one is universally better.
Diagnose a Prometheus cardinality explosion
Find offenders via TSDB stats and topk count by __name__, identify unbounded labels, then drop or aggregate them with relabeling. operating Prometheus at scale. just scaling memory without fixing label design.
What is distributed tracing in microservices?
A trace is a tree of spans tied by trace and span IDs, propagated via headers like W3C traceparent. observability across service boundaries. confusing tracing with plain logging or metrics.
Automate a canary release with a 1% 5xx error threshold
This tests wiring an SLO into an automated canary loop. A strong answer covers traffic splitting via a mesh or ingress, an analysis query to Prometheus for 5xx rate, and auto-promote or abort logic.
Koog 1.0 Stabilizes Kotlin AI Agent Framework
Koog 1.0 locks APIs for one year in JetBrains' Kotlin AI agent framework, adding local Android LiteRT inference and OpenTelemetry. JVM teams can build production agents without breaking changes. Adopt stable core first, then add beta features as needed.
How would you instrument and query P95 API latency by region?
This tests white-box latency instrumentation and safe cardinality for percentile aggregation. Strong answer: emit histograms by region, query P95 with histogram_quantile or a log percentile, and keep trace IDs in logs only.
How would you detect, mitigate, and penalize a noisy neighbor?
This tests multi-tenant fairness and governance. A strong answer covers tenant telemetry, graduated throttling, autoscaling with cost caps, and penalty tiers that curb abuse yet allow spikes. Red flag: hard limits before detection or graduated response.
Monitor p99 improvement from 500ms to 200ms and side effects
Tests systems-thinking on tail-latency instrumentation and metric tradeoffs. Strong answers cover histogram metrics, distributed tracing for fan-out bottlenecks, and guarding error rates, cost.
Describe leading vs lagging indicators with technical performance examples.
Tests your ability to distinguish predictors from outcomes. A strong answer defines causality, offers a leading metric like cache hit rate, and a lagging metric like P99 latency. Red flag: offering only business metrics or confusing activity with outcomes.
Get Observability bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.