The small files problem in data lakes

WHAT THIS TESTS This assesses whether you understand a subtle but common lake performance problem rooted in file count rather than data volume, and know the modern table formats that solve it.

A GOOD ANSWER COVERS The small files problem arises when a dataset is split across a very large number of tiny files. Every file imposes fixed costs: the engine must list it, open it, read its footer or metadata, and schedule a task, and on object stores listing itself is a relatively slow API call. When files are tiny, this per-file overhead dominates and the engine spends more effort on bookkeeping than on actual data, so queries and even simple directory listings slow dramatically and metadata services like a catalog or namenode get strained. Root causes include streaming or micro-batch ingestion that writes a small file every few seconds, many parallel writers each emitting a fragment, and over-partitioning that scatters little data across countless prefixes. The classic mitigation is compaction: periodically rewrite many small files into fewer large ones sized in the hundreds of megabytes. Modern table formats automate this and add more: Delta Lake, Apache Iceberg, and Apache Hudi provide a metadata layer with atomic commits, support OPTIMIZE or compaction operations and clustering, prune via statistics, and let you ingest in small increments while keeping the physical layout efficient.

COMMON WRONG ANSWERS Blaming total dataset size rather than file count and average file size. Ignoring listing overhead on object stores. Thinking more partitions always help, when over-partitioning causes the problem. Assuming a bigger cluster fixes it rather than compaction.

LIKELY FOLLOW-UPS What target file size do you aim for. How does Iceberg compaction work. How do you compact without disrupting readers.

ONE CONCRETE EXAMPLE A streaming job writes a 2 MB Parquet file every ten seconds, producing hundreds of thousands of tiny files a day, and queries crawl. Migrating the table to Iceberg and running scheduled compaction merges them into large files, restoring fast scans while still allowing frequent ingestion.