Design real-time usage-based billing data architecture

WHAT THIS TESTS: This question probes your ability to design distributed data systems where financial correctness is non-negotiable. The interviewer cares less about brand names and more about your grasp of fault tolerance, idempotency, stream processing semantics, and auditability at scale. They want to see how you balance real-time latency against exact monetary accuracy.

A GOOD ANSWER COVERS: First, an ingestion layer using an event log like Kafka configured for at-least-once delivery plus deterministic idempotency keys on every billable event so duplicates can be dropped without corrupting state. Second, a metering layer that aggregates raw events into time-bucketed usage totals using a columnar store like ClickHouse partitioned by tenant and event time to support millisecond-level queries. Third, a pricing and rating engine that maps aggregated meter values to billable line items through programmable rules such as tiered rates or credit wallet offsets. Fourth, immutable audit storage like S3 or a write-once ledger that retains raw events for seven-plus years to enable reconciliation and dispute resolution. Fifth, a reconciliation loop that continuously compares the sum of aggregated meter values against the raw event ledger to detect drift or late-arriving data within minutes not days.

COMMON WRONG ANSWERS: Claiming that Kafka exactly-once semantics alone prevent duplicate billing without mentioning idempotency keys or deduplication windows. Proposing a single relational database for both ingestion and aggregation at billion-event scale. Ignoring late-arriving events or suggesting full window reprocessing for every correction. Failing to separate the metering pipeline from the invoicing pipeline which creates coupling and outage risk. Omitting audit trails because you assume the database is the source of truth.

LIKELY FOLLOW-UPS: How would you handle a pricing plan change mid-month without reprocessing all historical events? What is your strategy when ClickHouse aggregation lags behind Kafka ingestion by ten minutes during a traffic spike? How do you guarantee auditability if a tenant disputes a bill six months later? Walk through the failure mode where the same event is published twice with different timestamps due to producer retries.

ONE CONCRETE EXAMPLE: Suppose an AI platform processes five hundred million API calls monthly. Each API gateway emits an event with a UUID, tenant ID, timestamp, and token count. The ingestion service writes these to Kafka with a retention of seven days. A stream processor reads the topic, deduplicates on UUID within a twenty-four-hour window, and writes aggregated hourly totals to ClickHouse. The pricing engine queries ClickHouse to apply tiered rates and subtract prepaid credits. Every hour a reconciliation job sums the raw Kafka events for that hour and compares the total to ClickHouse; any mismatch triggers an alert and a targeted backfill. Raw events are archived to S3 with checksums for long-term audit.

Source: Flexprice