Event-driven sync between billing and CRM

WHAT THIS TESTS The interviewer wants a design that keeps two services consistent through events while acknowledging that distributed messaging is at-least-once and unordered, so the consumer must be defensive.

A GOOD ANSWER COVERS When billing updates a tier, it writes the data change and a SubscriptionTierChanged event in the same database transaction using the outbox pattern, so you never have the dual-write problem where the DB commits but the event is lost. A relay publishes the outbox rows to a broker like Kafka or SNS/SQS. The CRM subscribes and applies the change idempotently: it dedupes on event id and uses a version number or timestamp so a stale, out-of-order event never overwrites a newer state. Add bounded retries with backoff, a dead-letter queue for poison messages, and monitoring on consumer lag. The result is eventual consistency between the systems with no synchronous coupling.

COMMON WRONG ANSWERS Having billing call the CRM synchronously, which couples availability and blocks billing when the CRM is down. Publishing the event after the transaction commits without an outbox, risking lost events. Assuming exactly-once delivery and skipping idempotency. Ignoring ordering, so an old event clobbers a new tier.

LIKELY FOLLOW-UPS How does the outbox pattern prevent dual writes? How do you handle a message that keeps failing? How do you reconcile if events are lost for hours?

ONE CONCRETE EXAMPLE A customer upgrades to Enterprise. Billing commits the row plus an outbox event atomically; the relay publishes it; the CRM consumer sees event id abc, version 5, updates the account to Enterprise, and records that it processed abc. A retried duplicate of abc is ignored, and an out-of-order version 4 event arriving late is dropped because the CRM already holds version 5. A periodic reconciliation job compares billing and CRM to catch anything the stream missed.