Optimize cost of a big-data analytics platform

WHAT THIS TESTS This checks breadth in cloud cost optimization: a senior answer attacks storage, compute, and the workload itself, not just instance size.

A GOOD ANSWER COVERS Storage: classify data by access frequency and apply lifecycle policies that transition cold data to cheaper infrequent-access or archive tiers and delete data past retention. Compress data and store it in columnar formats like Parquet or ORC, which shrink footprint dramatically versus raw JSON or CSV. Compute: batch jobs are interruption-tolerant, so run them on spot or preemptible instances for steep discounts, with checkpointing to survive reclamation. Right-size clusters to the job and use auto-scaling so capacity exists only while jobs run rather than idling. Workload and query optimization: partition and bucket data so jobs and queries read only relevant slices, and prefer columnar formats so engines scan fewer bytes, which directly lowers both runtime and, for scan-priced services, the bill. Add monitoring, tagging, and budgets to attribute and track spend.

COMMON WRONG ANSWERS Only switching to smaller instances. Ignoring storage tiering and retention. Leaving data uncompressed in row formats. Running interruption-tolerant batch on full-price on-demand. Forgetting that scanning less data is often the biggest lever.

LIKELY FOLLOW-UPS Why does columnar format cut cost for analytics? When are spot instances unsafe? How do partitioning and lifecycle policies work? How do you attribute cost to teams?

ONE CONCRETE EXAMPLE On AWS, raw logs land in S3 and a lifecycle policy moves data older than 30 days to S3 Glacier and expires it after a year. The daily Spark jobs run on an EMR cluster of spot instances with checkpointing, and the data is stored as partitioned Parquet so Athena and Spark scan only the needed date partitions, cutting both job runtime and per-scan charges substantially.