Parquet versus CSV for analytical data lakes

Why the comparison matters

Storage format silently determines query cost and speed in a data lake, where compute is often billed by bytes scanned. Choosing CSV for analytics can multiply both cost and runtime.

The full answer

Parquet is a columnar format, meaning values of one column are stored together rather than whole rows. Analytical queries typically touch a few columns out of many, so Parquet reads only those columns, projection pushdown, instead of scanning every byte as CSV requires. Because a column holds homogeneous values, Parquet compresses far better using encodings like dictionary and run-length plus a codec like Snappy, shrinking storage and bytes scanned. Parquet embeds a schema with explicit types, so there is no fragile type inference as with text CSV. It also stores per-row-group statistics such as min and max, letting engines skip entire row groups that cannot match a filter, predicate pushdown. CSV by contrast is row-oriented, plain text, untyped, usually uncompressed, and must be parsed and fully scanned, making it slow and expensive for large analytical workloads, though it remains human-readable and universally supported.

Trade-offs and caveats

CSV is simpler for tiny files, debugging, and interchange. Parquet is harder to inspect by eye and overkill for small data. Columnar layout does not help single-row point lookups or frequent full-row writes.

When it matters

Use Parquet for large analytical tables queried by engines like Spark, Athena, or BigQuery external tables, where scan cost dominates.

A concrete example

A hundred-column events table is queried for two columns filtered by date. In CSV the engine reads the entire file. In Parquet it reads only the two columns and skips row groups outside the date range using min/max stats, scanning a fraction of the bytes and finishing far faster and cheaper.