Keeping operator .status accurate under failures

What's really being asked

Whether you understand that status reflects observed reality at a point in time and that there is no atomic transaction spanning an external action and the status write.

The full answer

The core challenge is that updating an external service and writing the custom resource's status are two separate operations with no shared transaction. If the operator performs the external action, for example provisioning a database, and then crashes before writing status, the status is stale on restart; conversely a status write could succeed while the external call is uncertain. Network partitions compound this: the operator may be unable to observe the external service, so it cannot know the true state, and any cached value may be wrong. The robust approach is to treat reconcile as idempotent and source-of-truth-driven: each loop, actively re-observe the real external state rather than trusting cached or previously written status, then reconcile both the external resource and the status to match observed reality. Use the status subresource so spec and status writes do not clobber each other, model state with conditions and a observedGeneration field so consumers know which spec the status reflects, requeue on errors with backoff, and handle optimistic-concurrency conflict errors on status updates by retrying with a fresh object. Accept that status is eventually consistent.

The mistakes people make

Assuming the action and status update are atomic, trusting cached or last-written status as ground truth, ignoring update conflicts, or not re-observing external state after a restart.

What usually comes next

What is observedGeneration for? How do conditions help? How do you handle resourceVersion conflicts on status updates? How do you detect partitions versus real failures?

A concrete example

Your operator provisions an external cache, then crashes before setting status to Ready. On restart, instead of assuming anything from old status, reconcile queries the cache provider's API, sees it already exists and is healthy, and only then sets the Ready condition and observedGeneration. During a partition where the provider is unreachable, it sets a degraded condition and requeues rather than reporting a stale Ready.