The small files problem in data lakes

WHAT THIS TESTS This probes whether you understand that query performance on a lake depends heavily on physical file layout, and can propose concrete, distinct fixes.

WHAT IT IS The small files problem occurs when a dataset is split across an enormous number of very small files, kilobytes to a few megabytes each, instead of fewer large files. It commonly arises from streaming ingestion that flushes frequently, from over-partitioning, or from many parallel writers each emitting tiny outputs.

WHY IT DEGRADES PERFORMANCE Every file carries fixed overhead: the query engine must list it, open it, read its footer or metadata, and schedule a task for it. With millions of files this overhead dominates actual data reading. Object stores throttle and slow down on massive listings, and table metastores or catalogs strain under huge file counts. Columnar formats like Parquet also lose efficiency because each tiny file has its own header and limited row groups, defeating large sequential scans and predicate pushdown.

THREE DISTINCT STRATEGIES First, compaction: periodically run a job that bin-packs many small files into fewer large ones, targeting hundreds of megabytes each; table formats like Delta and Iceberg offer built-in OPTIMIZE or rewrite operations. Second, batch at write time: buffer streaming data and flush larger files less frequently, or use a tool like Kafka Connect or Firehose with size and time thresholds. Third, fix partitioning: choose coarser partition keys so each partition holds substantial data, avoiding high-cardinality partition columns that explode file counts.

LIKELY FOLLOW-UPS What target file size do you aim for. How does Iceberg or Delta compaction work. How does partitioning interact with file count. What is the tradeoff of compaction frequency.

ONE CONCRETE EXAMPLE A streaming job writes one Parquet file per minute per partition, producing millions of 200 KB files. Athena queries crawl. The team adds a nightly Spark compaction that rewrites each day's partition into 256 MB files and switches partitioning from hour to day, cutting query times dramatically.