Exactly-once semantics in stream processing

WHAT THIS TESTS This checks whether you understand that exactly-once is a guarantee about effects on state and output, not about physical message delivery, and can name how systems implement it.

A GOOD ANSWER COVERS Exactly-once processing semantics means every input event is reflected in the downstream state and output exactly one time, even when the system retries after failures. Contrast it with at-most-once, which may drop events, and at-least-once, which may process duplicates. The subtlety is that messages may be delivered multiple times over the network; the goal is that their effect is applied only once.

WHY IT IS HARD Failures can occur between any two steps. If a processor reads an event, writes output, then crashes before committing its input offset, on restart it replays the event and may double-apply the effect. Achieving exactly-once requires making the output write and the offset commit atomic, or making the effect safe to apply more than once.

MECHANISMS Idempotent sinks deduplicate using a stable key or sequence number, so reapplying the same record is a no-op, for example an upsert keyed by event id. Transactional writes bundle the output and the consumer offset into a single atomic transaction, as Kafka does with its transactional producer and read-committed isolation. Stream processors like Flink combine checkpointed operator state with a two-phase commit sink so state and output advance together. Producer idempotence prevents duplicate writes from retries within the broker.

LIKELY FOLLOW-UPS Difference between exactly-once delivery and processing. How do Kafka transactions work. How do Flink checkpoints give exactly-once. When is at-least-once with idempotency simpler.

ONE CONCRETE EXAMPLE A payment processor consumes charge events. Using an idempotent sink, each charge is upserted by a unique charge id, so even if the event is redelivered after a crash the customer is debited only once, achieving exactly-once effects without distributed transactions.