Machine Unlearning: From Data Deletion to Verifiable Forgetting

Model forgetting is harder than model training, and that's the uncomfortable truth behind every "right to be forgotten" request. When a user asks for their data to be deleted, most systems just remove the row from the database. The model, however, still carries traces of that data in its weights. Retraining from scratch on the full remaining dataset is the gold standard, but it's also the most expensive thing you can do in real-world ML. That's where machine unlearning comes in.

SISA (Sharded, Isolated, Sliced, Aggregated) is the most practical first stop. Instead of training one model on all data, you train multiple component models on disjoint shards, with incremental checkpoints per shard. To unlearn a data point, you only retrain the shard that contained it, and usually only from the last checkpoint before it was included. It's not a magic bullet—you still pay retraining cost per shard, and the aggregation can hurt accuracy if your shards are too small.

Gradient-based unlearning is more elegant but more fragile. The idea is to move the model's parameters away from the data you want to forget, usually by running gradient ascent on the loss for that specific set, followed by a few descent steps on the retained data to stabilize. Influence functions give you an approximate answer without retraining: estimate how much each training point changed the final weights, then subtract that. Both approaches have a major problem—they're approximate, and approximation error compounds.

Verification Is the Bottleneck

The tricky part is verification. How do you prove a model forgot something? Membership inference attacks (MIA) are the standard test—if the model no longer behaves as though it has seen the point (e.g., can't distinguish it from a random sample), you assume unlearning worked. But M