Why design ML pipeline steps to be idempotent?

WHAT THIS TESTS Whether you understand that pipeline steps will be re-executed, and whether you can design transforms so reruns are safe rather than corrupting data or producing duplicates.

A GOOD ANSWER COVERS Idempotency means that executing a step once or many times with the same input yields the same output and the same end state in any external system, with no accumulating side effects. It is desirable because failures and retries are normal in distributed pipelines: a worker crashes mid-write, an orchestrator retries a task, or you backfill a date range that partially ran before. If steps are idempotent, all of these are safe and the result is deterministic and reproducible. To design an idempotent feature engineering step, make the transformation itself deterministic by avoiding wall-clock timestamps, random seeds without fixing them, and ordering-dependent operations. Then make the write idempotent: partition outputs by a stable key such as the date or entity id, and overwrite that partition or upsert by primary key rather than appending. That way a rerun replaces the slice it owns instead of duplicating rows.

COMMON WRONG ANSWERS Using append-only inserts means each retry duplicates rows. Embedding the current timestamp or an unfixed random seed in features makes outputs differ run to run, breaking reproducibility. Relying on auto-increment ids tied to insertion order is also non-idempotent.

LIKELY FOLLOW-UPS How do you make an external API call idempotent? How do you handle idempotency across partial failures within a single step? How does this interact with exactly-once versus at-least-once delivery?

ONE CONCRETE EXAMPLE A daily feature job computes per-user aggregates for date D. Instead of appending rows, it writes to a partition keyed by D and overwrites it. If the orchestrator retries the task after a crash, the partition is simply rewritten with identical values, so no duplicate or stale rows survive and a backfill of last month reproduces the exact same table.