Kubernetes Audit Logging: Your Cluster's Black Box Recorder

Why it exists

A Kubernetes cluster is a dynamic system with many actors—users, controllers, service accounts—constantly making changes. Without a record, you can't investigate security incidents, satisfy compliance auditors, or debug why a critical application suddenly disappeared. Audit logs provide this essential, chronological record of all cluster activity.

The mental model

Think of the Kubernetes API server as the single, guarded entrance to your cluster's state. Audit logging is the security camera system installed at that entrance. It records every person (user) and robot (controller) that enters, what they were carrying (the request body), and what they did inside (the API verb like 'create', 'delete', 'patch').

How it works

You provide the kube-apiserver with an audit policy file. This policy contains rules that specify what to log. Each rule matches requests based on stages of execution, users, resources (pods, secrets), and verbs (get, delete). For each matching rule, you specify an audit level: None (ignore), Metadata (log who, when, where, but not what), Request (log metadata and the request payload), or RequestResponse (log everything). The API server then sends these generated audit events to a configured backend, typically a log file or a remote webhook endpoint.

When to use it

Always enable it in any cluster that runs meaningful workloads. It is critical for security incident response, allowing you to trace an attacker's actions. It's required for most compliance frameworks like PCI DSS, HIPAA, and SOC 2. It's also a powerful tool for debugging, helping you understand why a controller is behaving unexpectedly or what automated process is modifying a resource.

When not to use it

It's rarely a good idea to disable it completely. However, you might use a minimal policy on a short-lived, local development cluster to save resources. The more important question is what not to log. You must be selective to avoid creating a uselessly large and expensive dataset, which is a common failure mode.

One canonical example

A common and effective policy is to log all requests that modify data (create, update, delete, patch) at the RequestResponse level. For high-volume, read-only requests (get, list, watch), you might log them at the Metadata level or not at all, especially for internal system components. This strikes a balance between capturing critical changes and managing log volume. For example, you always want to know who created a Secret, but you may not need to log every time the kubelet checks a Pod's status.