Scalable multi-modal data quality pipeline

WHAT THIS TESTS Whether you can architect a distributed, automated, and observable data quality pipeline for heterogeneous, noisy, multi-modal data at foundation-model scale.

A GOOD ANSWER COVERS Use a staged pipeline on a distributed engine such as Spark or Ray, writing intermediate results to an object-store data lake so stages are restartable. Stage one is ingestion and schema or integrity validation: reject corrupt files, verify checksums, and normalize formats. Stage two is modality-specific filtering: drop unplayable video and audio, enforce minimum resolution and sample rate, detect and filter language for text, and run perceptual checks. Stage three is deduplication: exact dedup by content hash and near-duplicate removal using embeddings or MinHash, which is critical because duplicates inflate compute and skew training. Stage four is safety and compliance: strip or flag PII, filter toxicity, and respect licensing and consent. Stage five is quality scoring: assign per-sample quality scores, quarantine low-quality shards rather than silently dropping them, and emit batch-level metrics.

OBSERVABILITY AND AUTOMATION Every batch records counts dropped per rule, distribution stats, and drift versus prior batches; thresholds gate promotion so a bad source cannot poison training unnoticed.

COMMON WRONG ANSWERS A single one-pass script, skipping deduplication, dropping data with no audit trail, or applying one cleaning rule across all modalities.

LIKELY FOLLOW-UPS How you tune dedup similarity thresholds, balance recall of bad data against false drops, and keep human review in the loop.

ONE CONCRETE EXAMPLE A video source uploads near-identical clips. Embedding-based near-dup detection collapses them to one, schema validation rejects truncated files, a PII pass blurs faces and removes captions with personal data, and the batch report shows twelve percent dropped, prompting a check of that low-quality source before training.