Idempotency in infrastructure provisioning scripts

WHAT THIS TESTS: Whether you can build provisioning automation that survives retries, network blips, and reruns without creating duplicate or corrupted resources.

A GOOD ANSWER COVERS: Idempotency means running the operation once or many times yields the same final state. If a network timeout makes you unsure whether a create succeeded, you can safely retry without spinning up a second instance. Technique one is declarative desired-state reconciliation: describe the target state, read the current state, compute the diff, and apply only what is missing. Tools like Terraform and Kubernetes controllers work this way. Technique two is idempotency keys: the client generates a unique token per logical operation and sends it with each request; the server records the token and returns the original result for duplicates instead of acting again. A simpler variant is read-before-write or an upsert that creates only if absent.

COMMON WRONG ANSWERS: Saying idempotency just means retrying, claiming POST is inherently safe, or relying on naming alone without checking existing state, which races under concurrency.

LIKELY FOLLOW-UPS: How long do you store idempotency keys? How do you handle partial failures mid-provision? What about concurrent runs racing on the same resource? How does this interact with eventual consistency in the cloud API?

ONE CONCRETE EXAMPLE: A script provisions a virtual machine. With desired-state reconciliation, it lists existing VMs by a stable tag, finds none, and creates one; on rerun it finds the VM and does nothing. Alternatively, it sends the create call with an idempotency key derived from a request UUID; if a timeout triggers a retry, the cloud API recognizes the same key and returns the existing VM rather than launching a duplicate, so a flaky network never doubles your fleet or your bill.