Design a cryptographically verifiable ML audit trail from dataset to deployment

WHAT THIS TESTS: This question tests whether you understand cryptographic provenance and tamper-evident systems beyond simple version control. The interviewer wants to see if you can design an ML supply chain where no single actor, including a malicious insider, can alter the lineage from an approved dataset to a deployed model without detection. It also checks your grasp of separation of duties, reproducibility, and the difference between confidentiality and integrity.

A GOOD ANSWER COVERS: A strong answer hits four things in order. First, content addressing for all inputs: the dataset, feature engineering code, and training configuration are hashed or Merkleized so any bit-level change produces a different root digest. Second, an append-only provenance log that binds these input hashes to the training run metadata, including dependency versions, hardware identifiers, and hyperparameters, and captures the resulting model artifact hash. Third, cryptographic signing: the model is signed at rest by a build-time key held in an HSM or secure enclave, and the serving infrastructure verifies that signature before loading weights. Fourth, separation of duties: the audit log operator is distinct from the training infrastructure team, preventing collusion to rewrite history.

COMMON WRONG ANSWERS: Candidates often confuse encryption with integrity, proposing to encrypt the model at rest without any mechanism to prove it came from a specific training run. Another red flag is suggesting a public blockchain as the default audit layer without analyzing write throughput, cost, or data privacy; for most regulated enterprises, a centralized append-only log with cryptographic witnesses and multi-party attestation is sufficient and more practical. A third mistake is ignoring code and configuration provenance, focusing only on the dataset and final weights while omitting the training script version or random seed.

LIKELY FOLLOW-UPS: The interviewer may ask how you handle non-determinism in GPU floating point operations, how you rotate signing keys without breaking historical verification, or how you prove to the regulator that the audit log itself has not been tampered with. They might also probe whether the verification happens continuously at inference time or only at deployment, and how you handle model updates or A-B testing artifacts.

ONE CONCRETE EXAMPLE: Imagine a healthcare imaging model. The approved dataset is stored in a content-addressable store with a SHA-256 root hash published to an internal transparency log. A CI pipeline runs the training job inside a confidential computing environment, producing a model file and a signed attestation that lists the dataset hash, Git commit, hyperparameters, and output model hash. The model artifact and attestation are stored in immutable object storage. At deployment, the serving platform refuses to load the model unless the attestation signature is valid, the dataset hash matches the regulator-approved entry, and the transparency log contains a matching inclusion proof. Six months later, an auditor can replay the exact hash chain and confirm no substitution occurred.