Guarantee at-least-once delivery for a critical analytics event?
Tests reliable messaging patterns to avoid dual-write issues. Propose the Transactional Outbox pattern: atomically write business data and the event to a DB outbox table. A separate relay process then sends the event.
What's really being asked
This question tests your understanding of atomicity and data consistency in distributed systems. The core challenge is the "dual-write problem": how to atomically update your primary database AND send a message to an external system (like an analytics service or message broker) without using distributed transactions (2PC), which are often unsupported or undesirable. The interviewer is looking for a specific, robust pattern, not just simple retry logic.
The full answer
A strong answer outlines the Transactional Outbox pattern. It should cover four key points in order. First, acknowledge the core problem: a service crash between the database commit and the analytics API call can lead to lost events. Second, propose storing the event payload in a dedicated "outbox" table within the SAME database as the primary business data. Third, explain that this write happens within the SAME atomic transaction as the business logic (e.g., UPDATE accounts; INSERT INTO outbox_events; COMMIT;). This guarantees that if the transaction succeeds, the event is durably stored. Fourth, describe a separate, asynchronous "message relay" process. This process is responsible for reading unprocessed events from the outbox table and reliably sending them to the analytics backend. After successful delivery, it marks the event as processed.
The mistakes people make
The most common mistake is proposing a naive solution. This includes: 1) Firing the event before the database transaction commits (if the commit fails, you've sent a phantom event). 2) Firing the event after the commit in a simple try/catch block (if the service crashes after the commit but before the send, the event is lost forever). 3) Relying solely on client-side instrumentation, which is irrelevant for a critical server-side financial transaction. A senior candidate should immediately recognize this as a database atomicity problem, not just a network reliability problem.
What usually comes next
Be ready for follow-ups on the message relay implementation. How would you build it? Two common patterns are a polling publisher that queries the outbox table, or a transaction log tailing process that reads from the DB's change log. Another follow-up is: "What if the relay sends an event but crashes before marking it as sent?" This leads to duplicate delivery. The answer is that the downstream consumer (the analytics backend) MUST be designed to be idempotent, for example by de-duplicating events based on a unique event ID.
A concrete example
For a $100 payment, the PaymentsService starts a transaction. It updates the user's balance in the accounts table and, in the same transaction, inserts a JSON payload for the payment_successful event into the transactional_outbox table with a status of 'PENDING'. The transaction commits. A separate OutboxRelay process runs every 5 seconds, querying for 'PENDING' events. It finds the event, sends it to the analytics service's HTTP endpoint, and upon receiving a 200 OK, updates the event's status in the outbox table to 'SENT'. If the service had crashed after the DB commit, the relay would simply find and send the event on its next run.
Interview question
A service updates a database and must then send a critical event. How can you best ensure the event is reliably sent if the database update succeeds, even if the service crashes?
- a.Commit the database transaction, then immediately call the event service in a try/catch block to handle network failures.
- b.Write the event to an 'outbox' table within the same database transaction as the primary update. A separate process then sends events from this table.Correct
- c.Use a distributed transaction (2PC) to atomically commit changes to both the primary database and the remote event service.
- d.Send the event first, and only commit the database transaction if the send is successful.
Why? this is the answer
D is correct because writing the event and business data in one atomic transaction guarantees the event is durably saved if the business logic succeeds. B is a common but flawed approach; the service could crash after the commit but before sending the event, losing it forever.
Just read this? Test yourself on what you have been reading.
Read the original → microservices.io
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on system design — each one lists the topics its interview covers.
See open roles