What causes client order_completed events to diverge from backend records?

WHAT THIS TESTS: This question tests whether you can reason about distributed telemetry pipelines and distinguish between lossy client-side instrumentation and durable server-side state. Senior engineers are expected to avoid treating either source as absolute truth and to apply systematic debugging across network, application, and data layers.

A GOOD ANSWER COVERS: First, establish direction and magnitude. If client events exceed backend orders, investigate network drops between client and ingestion endpoint, ad blockers or tracking prevention browsers, mobile app crashes before event flush, duplicate events from retry storms without idempotency keys, and clock skew causing events to fall outside processing windows. If backend orders exceed client events, look at server-side retries creating duplicate database rows, batch jobs that backfill orders without corresponding events, and events intentionally suppressed for fraud or test accounts. Second, propose a phased investigation. Phase one is quantification: run a time-bound join on user ID and order ID across both systems, compute the daily delta, and segment by platform, region, and app version. Phase two is validation: check for duplicate event IDs in the event stream, verify timezone and attribution window alignment, and inspect sampling logic. Phase three is tracing: pick a sample of orphaned events and follow them through the ingestion API, message queue, and ETL jobs to identify the exact drop point. Third, discuss fixes. Implement idempotency keys on order_completed events, move critical event logging server-side where possible, add out-of-band reconciliation jobs that alert when deltas exceed a threshold, and use exactly-once semantics in your stream processor.

COMMON WRONG ANSWERS: Treating the client log as ground truth is a major red flag. Blaming the gap on user error or fraud without data is another. Proposing to add more client-side logging without first measuring the gap direction suggests a spray-and-pray approach. Recommending a full pipeline rewrite before isolating the failure point signals poor judgment under ambiguity.

LIKELY FOLLOW-UPS: How would you design an idempotency key scheme that survives client retries? What reconciliation latency is acceptable for financial reporting versus marketing analytics? How do you handle offline-first mobile apps where events arrive hours later? Would you move order tracking entirely server-side, and what are the trade-offs?

ONE CONCRETE EXAMPLE: Suppose the mobile app fires order_completed when the user sees a confirmation screen, but the backend only commits the order after a third-party payment webhook succeeds. If the user closes the app immediately after purchase, the event may fire while the webhook is still pending, and if the payment fails, the backend rolls back the order while the event remains in the warehouse. The fix is to anchor the event to the backend state change rather than the UI confirmation, or to emit a corrected event if the transaction aborts.

Source: improvado.io