How do you guarantee at-least-once event delivery for a financial transaction?

What's really being asked

This question probes your ability to guarantee atomicity between a local database transaction and an external side effect without using distributed transactions such as two-phase commit. The interviewer cares that you understand the dual-write problem, where a service crash or network partition can leave the database and analytics backend inconsistent.

The full answer

First, explain that the application must write the analytics event to an outbox table stored in the same database as the business data, using a single local transaction. Second, describe a separate message relay process that polls the outbox or tails the transaction log, publishes events to the analytics backend, and retries on failure until it receives an acknowledgment. Third, note that because the relay might crash after publishing but before deleting or marking the message as sent, the analytics consumer must be idempotent, typically by tracking processed event IDs. Fourth, mention that ordering is preserved because messages are read from the outbox in insertion order and published sequentially.

The mistakes people make

A red flag is suggesting a fire-and-forget HTTP POST from the application thread immediately after the database commits, because a process crash between commit and send loses the event forever. Another red flag is proposing a saga or 2PC across the database and the analytics backend, since most analytics systems do not support XA transactions and the coupling is undesirable. Simply using a message queue without the outbox also fails if the queue write is not atomic with the business transaction.

What usually comes next

The interviewer may ask how to implement the message relay, comparing polling publisher versus transaction log tailing. They might ask how to handle high throughput, which could lead to discussion of batching publishes or using a separate CDC stream like Debezium. They may also ask how to guarantee ordering across multiple service instances updating the same aggregate, or how to prune the outbox table once messages are confirmed sent.

A concrete example

Imagine a payment service processing a 500 dollar transfer. Inside the same PostgreSQL transaction that debits the sender and credits the receiver, the service inserts a row into an analytics_outbox table containing a JSON payload with the transaction ID, amount, and timestamp. A background worker queries the outbox every second, publishes each row to a Kafka topic, and updates the row status to SENT only after Kafka acknowledges the write. If the worker restarts, it resumes from the earliest unsent row, ensuring at-least-once delivery. The analytics service consumes the topic and drops duplicates by checking a processed_event_ids table keyed by the event ID.