Design real-time usage-based billing data architecture
Tests whether you can guarantee exactly-once billing at scale without data loss. Strong answers cover idempotent Kafka ingestion, ClickHouse aggregation, reconciliation, and audit trails. Red flag: claiming exactly-once instead of at-least-once dedup.
What's really being asked
This question probes your ability to design distributed data systems where financial correctness is non-negotiable. The interviewer cares less about brand names and more about your grasp of fault tolerance, idempotency, stream processing semantics, and auditability at scale. They want to see how you balance real-time latency against exact monetary accuracy.
The full answer
First, an ingestion layer using an event log like Kafka configured for at-least-once delivery plus deterministic idempotency keys on every billable event so duplicates can be dropped without corrupting state. Second, a metering layer that aggregates raw events into time-bucketed usage totals using a columnar store like ClickHouse partitioned by tenant and event time to support millisecond-level queries. Third, a pricing and rating engine that maps aggregated meter values to billable line items through programmable rules such as tiered rates or credit wallet offsets. Fourth, immutable audit storage like S3 or a write-once ledger that retains raw events for seven-plus years to enable reconciliation and dispute resolution. Fifth, a reconciliation loop that continuously compares the sum of aggregated meter values against the raw event ledger to detect drift or late-arriving data within minutes not days.
The mistakes people make
Claiming that Kafka exactly-once semantics alone prevent duplicate billing without mentioning idempotency keys or deduplication windows. Proposing a single relational database for both ingestion and aggregation at billion-event scale. Ignoring late-arriving events or suggesting full window reprocessing for every correction. Failing to separate the metering pipeline from the invoicing pipeline which creates coupling and outage risk. Omitting audit trails because you assume the database is the source of truth.
What usually comes next
How would you handle a pricing plan change mid-month without reprocessing all historical events? What is your strategy when ClickHouse aggregation lags behind Kafka ingestion by ten minutes during a traffic spike? How do you guarantee auditability if a tenant disputes a bill six months later? Walk through the failure mode where the same event is published twice with different timestamps due to producer retries.
A concrete example
Suppose an AI platform processes five hundred million API calls monthly. Each API gateway emits an event with a UUID, tenant ID, timestamp, and token count. The ingestion service writes these to Kafka with a retention of seven days. A stream processor reads the topic, deduplicates on UUID within a twenty-four-hour window, and writes aggregated hourly totals to ClickHouse. The pricing engine queries ClickHouse to apply tiered rates and subtract prepaid credits. Every hour a reconciliation job sums the raw Kafka events for that hour and compares the total to ClickHouse; any mismatch triggers an alert and a targeted backfill. Raw events are archived to S3 with checksums for long-term audit.
Interview question
Why is Kafka configured for at-least-once delivery with deterministic idempotency keys instead of relying solely on Kafka exactly-once semantics?
- a.Because idempotency keys let the metering layer skip the stream processor and write directly to ClickHouse for faster aggregation.
- b.Because producer retries can emit the same logical event with different timestamps, which exactly-once semantics do not always prevent, requiring explicit deduplication.Correct
- c.Because at-least-once delivery guarantees zero message loss, whereas exactly-once semantics are vulnerable to consumer rebalancing failures.
- d.Because exactly-once semantics add prohibitive latency and reduce throughput when processing billions of billable events daily.
Why? this is the answer
The card explicitly warns that relying solely on Kafka exactly-once semantics is a red flag because producer retries can publish duplicate events with altered timestamps, making deterministic idempotency keys essential for correct deduplication. Option D is tempting because streaming performance is a common concern, but the architecture is driven by monetary correctness rather than latency.
Just read this? Test yourself on what you have been reading.
Read the original → flexprice.io
- #system design
- #data architecture
- #distributed systems
- #billing
- #kafka
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on system design — each one lists the topics its interview covers.
See open roles