Exactly-once semantics in stream processing
understanding delivery guarantees and effects.
exactly-once means each event affects state once despite retries, it is hard because of failures between processing and committing, and you achieve it via idempotency or atomic…
What's really being asked
This checks whether you understand that exactly-once is a guarantee about effects on state and output, not about physical message delivery, and can name how systems implement it.
The full answer
Exactly-once processing semantics means every input event is reflected in the downstream state and output exactly one time, even when the system retries after failures. Contrast it with at-most-once, which may drop events, and at-least-once, which may process duplicates. The subtlety is that messages may be delivered multiple times over the network; the goal is that their effect is applied only once.
Why it is hard
Failures can occur between any two steps. If a processor reads an event, writes output, then crashes before committing its input offset, on restart it replays the event and may double-apply the effect. Achieving exactly-once requires making the output write and the offset commit atomic, or making the effect safe to apply more than once.
Mechanisms
Idempotent sinks deduplicate using a stable key or sequence number, so reapplying the same record is a no-op, for example an upsert keyed by event id. Transactional writes bundle the output and the consumer offset into a single atomic transaction, as Kafka does with its transactional producer and read-committed isolation. Stream processors like Flink combine checkpointed operator state with a two-phase commit sink so state and output advance together. Producer idempotence prevents duplicate writes from retries within the broker.
What usually comes next
Difference between exactly-once delivery and processing. How do Kafka transactions work. How do Flink checkpoints give exactly-once. When is at-least-once with idempotency simpler.
A concrete example
A payment processor consumes charge events. Using an idempotent sink, each charge is upserted by a unique charge id, so even if the event is redelivered after a crash the customer is debited only once, achieving exactly-once effects without distributed transactions.
Interview question
What does 'exactly-once' actually guarantee in a streaming pipeline?
- a.Messages are never duplicated because retries are disabled entirely
- b.Every event is processed within a fixed latency budget
- c.Each event affects downstream state and output exactly once despite retriesCorrect
- d.Each message is physically transmitted over the network exactly one time
Why? this is the answer
Exactly-once is about effects: an event is reflected in state and output once even if delivered multiple times, achieved via idempotency or transactional commits. The network may still deliver duplicates.
Just read this? Test yourself on what you have been reading.
Read the original → conduktor.io
- #exactly-once
- #streaming
- #kafka
- #idempotency
- #data-engineering
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles