Maximizing object-store throughput for small files

WHAT THIS TESTS The interviewer probes whether you understand object-store internals, request overhead and key-based partitioning, well enough to fix a small-file throughput problem.

A GOOD ANSWER COVERS Identify the two real bottlenecks. First, per-request overhead: each PUT is a separate authenticated HTTP request, so millions of tiny objects mean millions of round trips, and request rate, not raw bandwidth, becomes the limit. Second, key distribution: object stores partition by key prefix, and historically a sequential or timestamp-leading prefix concentrated traffic on one partition, creating a hotspot. Re-architect on both fronts. Spread writes across many high-entropy prefixes so the service parallelizes across partitions. Batch many small files into larger aggregate objects, for example bundling into archives or columnar files, drastically cutting request count and storage overhead. Parallelize the uploads with many concurrent workers and connections rather than a single serial writer, and use multipart upload for the larger aggregated objects. Buffer and aggregate at the edge before writing so the object store sees fewer, larger, well-distributed writes.

COMMON WRONG ANSWERS Concluding the service is simply slow and asking for a quota bump, ignoring per-object overhead. Writing all keys under one sequential prefix, hotspotting a partition. Uploading serially with one client. Not batching, so the request count stays in the millions.

LIKELY FOLLOW-UPS How does prefix-based partitioning cause hotspots and how do you avoid them. What is the cost and trade-off of batching small files. How does multipart upload help. How would you buffer and aggregate upstream, for example with a streaming layer.

ONE CONCRETE EXAMPLE An IoT pipeline writes one tiny object per device reading, millions per hour, all keyed by an incrementing timestamp, and throughput stalls. The team adds a streaming buffer that aggregates readings into hourly columnar files per device shard, writes them under hashed prefixes spread across partitions, and uploads with a pool of concurrent workers using multipart. Request count drops by orders of magnitude and throughput meets the requirement.