Design a highly available entitlements service with caching

What's really being asked

The interviewer wants to see if you understand that entitlements sit on the critical path of every user request, so latency and availability are paramount. They are testing your ability to reason about distributed caching, cache invalidation, handling time-bound state like temporary grants, and maintaining consistency across regions without turning the database into a bottleneck.

The full answer

First, a multi-tier cache strategy. Use a local in-process cache like Caffeine for sub-millisecond hot lookups, backed by a distributed Redis cluster with TTL for cross-instance consistency. Second, a proactive invalidation pipeline. Writes to the entitlement store should publish invalidation events over a message bus so both local and remote caches evict stale entries immediately rather than waiting for TTL. Third, separation of hot and cold paths. Static plan features can be cached aggressively; temporary grants need short TTLs or event-sourced append-only logs so they do not pollute the main cache with churn. Fourth, resilience patterns. Deploy circuit breakers around cache misses to prevent a cold cache from overwhelming the primary database, and use cache warming after deploys. Fifth, consistency model clarity. Acknowledge that entitlements can tolerate brief eventual consistency for reads but require atomic writes, so use write-through caching with optimistic locking on the primary store.

The mistakes people make

Proposing a single shared Redis instance with no invalidation strategy and relying solely on TTL. This creates stale data windows where a user pays for a plan but cannot access features, or worse, retains access after cancellation. Another red flag is ignoring the thundering herd problem; if the cache expires and thousands of requests hit the database simultaneously, the service collapses. Suggesting strong consistency everywhere is also a mistake because it forces cross-region coordination on every read and destroys latency.

What usually comes next

How do you handle a cache invalidation event that gets dropped? How do you model temporary grants without degrading cache hit rates? What happens when the primary database is partitioned but the cache is still serving data, and do you allow or deny access? How do you rollout a new plan definition without restarting the service?

A concrete example

Imagine a streaming platform where users upgrade from Basic to Premium mid-session. The entitlement service receives the plan change, writes to PostgreSQL, and emits an invalidation event to Redis pub-sub and Kafka. Edge nodes consume the event and evict the local Caffeine entry. The next request from that user misses local cache, hits Redis which also misses, then queries PostgreSQL and repopulates both layers. For a temporary grant like a 48-hour sports pass, the system writes an append-only event with an explicit expiration timestamp; the cache stores it with a TTL matching the expiration, so no explicit invalidation is needed when it lapses.