Data lineage and machine unlearning for a fine-tuned LLM

WHAT THIS TESTS Whether you can build provenance from raw source to model output and whether you understand that influence baked into weights is not trivially removable.

A GOOD ANSWER COVERS Lineage has three layers. Source layer: register each proprietary dataset as an immutable, versioned artifact with a content hash and license metadata. Transform layer: capture every preprocessing step as code-versioned, reproducible operations so any training example can be traced back to its raw origin. Training layer: record which dataset versions and example ranges fed each checkpoint, stored as a manifest linked to model versions in a registry. This lets you answer which sources influenced model v3. For unlearning, be honest about cost. The gold standard is retraining from the last clean checkpoint without the offending source. To make that affordable, use sharded, isolated training such as SISA, where data is split into shards and slices so removal only forces retraining of the affected shards. Approximate unlearning via targeted gradient ascent or influence-function corrections is faster but needs verification that the source's effect is actually gone, for example via membership-inference or canary tests.

COMMON WRONG ANSWERS Claiming you can just delete the rows and the model forgets. Treating lineage as only storing raw files without transform reproducibility. Ignoring that approximate unlearning must be verified. Forgetting license and consent metadata required by regulation.

LIKELY FOLLOW-UPS How do you verify unlearning succeeded? What is the cost difference between SISA and full retraining? How do canaries detect memorized data? How do you handle a source that is entangled with others?

ONE CONCRETE EXAMPLE Dataset D2 must be removed after a license revocation. The manifest shows D2 trained shards 3 and 7 of a SISA setup. You retrain only those shards from their checkpoints and recombine, far cheaper than full retraining. To verify, you run membership-inference and canary-string probes confirming D2-specific content is no longer recoverable, then record the unlearning event in the lineage log for the audit trail.