The Three Pillars of Observability
Observability isn't one tool; it's a three-legged stool of metrics, logs, and traces. Metrics give the 'what' (CPU is high), logs the 'why' (an error loop), and traces the 'where' (which service is slow). The footgun is treating them as separate silos.
WHY IT EXISTS: In complex, distributed systems like those managed by Kubernetes, a single failure can cascade in unpredictable ways. Simply knowing "something is broken" is not enough. Observability provides the tools to ask arbitrary questions about your system's state without having to ship new code to answer them.
THE MENTAL MODEL: Think of debugging a system like a doctor diagnosing a patient. Metrics are the vital signs (temperature, heart rate). Logs are the patient's detailed account of what happened ("I felt dizzy after eating..."). Traces are like an MRI, showing the entire pathway of blood flow to find the exact blockage. You need all three for a confident diagnosis.
HOW IT WORKS: The three pillars work together. First, metrics (like CPU usage, request latency, error rates) are collected as time-series data, often visualized in dashboards. They alert you to a problem. Second, logs provide detailed, timestamped records of specific events, like errors or transactions, giving context to the metric spike. Third, distributed traces follow a single request as it travels through multiple services, pinpointing bottlenecks or failures in the chain.
WHEN TO USE IT: Use the three pillars to debug production issues in any non-trivial application, especially microservices. They are essential for understanding performance bottlenecks, tracking down the root cause of errors, and monitoring overall system health. If you can't answer "why is it slow?" you need better observability.
WHEN NOT TO USE IT: The full trifecta might be overkill for simple, monolithic applications running on a single server where a stack trace and server logs are sufficient. The overhead of instrumenting for distributed tracing, in particular, may not be justified for very simple systems. The main footgun is collecting data without a plan to use or correlate it, leading to high storage costs with little insight.
ONE CANONICAL EXAMPLE: A Prometheus metric alert fires for high latency on the checkout service. You check the logs for that service around the alert time and see a spike in "database connection timeout" errors. You then pull up a distributed trace for a slow checkout request and see that 95% of the time is spent waiting for a response from the inventory database, a separate microservice. You've used all three pillars to pinpoint the root cause.
Read the original → kubernetes.io
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.