• Corpus ID: 232134970

Remember What You Want to Forget: Algorithms for Machine Unlearning

@inproceedings{Sekhari2021RememberWY,
  title={Remember What You Want to Forget: Algorithms for Machine Unlearning},
  author={Ayush Sekhari and Jayadev Acharya and Gautam Kamath and Ananda Theertha Suresh},
  booktitle={NeurIPS},
  year={2021}
}
We study the problem of unlearning datapoints from a learnt model. The learner first receives a dataset S drawn i.i.d. from an unknown distribution, and outputs a model b w that performs well on unseen samples from the same distribution. However, at some point in the future, any training datapoint z 2 S can request to be unlearned, thus prompting the learner to modify its output model while still ensuring the same accuracy guarantees. We initiate a rigorous study of generalization in machine… 
On the Necessity of Auditable Algorithmic Definitions for Machine Unlearning
TLDR
Unlearning is only well-defined at the algorithmic level, where an entity’s only possible auditable claim to unlearning is that they used a particular algorithm designed to allow for external scrutiny during an audit.
Unrolling SGD: Understanding Factors Influencing Machine Unlearning
TLDR
This work identifies verification error, i.e., the `2 difference between the weights of an approximately unlearned and a naively retrained model, as an approximate unlearning metric that should be optimized for as it subsumes a large class of other metrics.
Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teacher
TLDR
This work proposes a novel machine unlearning method by exploring the utility of competent and incompetent teachers in a student-teacher framework to induce forgetfulness, and introduces a zero retrain forgetting (ZRF) metric to evaluate the un learning method.
Deep Unlearning via Randomized Conditionally Independent Hessians
TLDR
Compared to alternatives, L-CODEC makes approximate unlearning possible in settings that would otherwise be infeasible, including vision models used for face recognition, person reidentification and NLP models that may require unlearning samples identified for exclusion.
Zero-Shot Machine Unlearning
TLDR
This paper introduces the novel problem of zero-shot machine unlearning that caters for the extreme but practical scenario where zero original data samples are available for use, and proposes two novel solutions based on error minimizing-maximizing noise and gated knowledge transfer.
Evaluating Inexact Unlearning Requires Revisiting Forgetting
TLDR
It is empirically show that two simple un learning methods, exact-unlearning and catastrophic-forgetting the final k layers of a network, outperforms prior unlearning methods when scaled to large deletion sets.
Hard to Forget: Poisoning Attacks on Certified Machine Unlearning
TLDR
It is demonstrated how an attacker can exploit this oversight, highlighting a novel attack surface introduced by machine unlearning, and an attacker aiming to increase the computational cost of data removal is considered.
Vertical Machine Unlearning: Selectively Removing Sensitive Information From Latent Feature Space
TLDR
This paper investigates a vertical unlearning mode, aiming at removing only sensitive information from latent feature space, and introduces a ground-breaking solution based on representation detachment, where the task-related information is encouraged to retain while the sensitive information is progressively forgotten.
Adaptive Machine Unlearning
TLDR
This paper shows in theory how prior work for non-convex models fails against adaptive deletion sequences, and uses this intuition to design a practical attack against the SISA algorithm of Bourtoule et al.
Efficient Attribute Unlearning: Towards Selective Removal of Input Attributes from Feature Representations
TLDR
The proposed unlearning paradigm of selectively removing input attributes from feature representations which is named ‘attribute unlearning’ builds a foundation for future machine unlearning system and will become an essential component of the latest privacy-related legislation.
...
...

References

SHOWING 1-10 OF 39 REFERENCES
Machine Unlearning
TLDR
This work introduces SISA training, a framework that decreases the number of model parameters affected by an unlearning request and caches intermediate outputs of the training algorithm to limit thenumber of model updates that need to be computed to have these parameters unlearn.
Making AI Forget You: Data Deletion in Machine Learning
TLDR
This paper proposes two provably efficient deletion algorithms which achieve an average of over 100X improvement in deletion efficiency across 6 datasets, while producing clusters of comparable statistical quality to a canonical k-means++ baseline.
Variational Bayesian Unlearning
TLDR
This paper studies the problem of approximately unlearning a Bayesian model from a small subset of the training data to be erased using the variational inference (VI) framework and proposes two novel tricks to tackle this challenge.
Towards Making Systems Forget with Machine Unlearning
TLDR
This paper presents a general, efficient unlearning approach by transforming learning algorithms used by a system into a summation form, and applies to all stages of machine learning, including feature selection and modeling.
When Machine Unlearning Jeopardizes Privacy
TLDR
This paper proposes a novel membership inference attack that leverages the different outputs of an ML model's two versions to infer whether a target sample is part of the training set of the original model but out of theTraining set of a corresponding unlearned model.
SGD: The Role of Implicit Regularization, Batch-size and Multiple-epochs
TLDR
This paper considers the problem of SCO and explores the role of implicit regularization, batch size and multiple epochs for SGD, and extends the results to the general learning setting by showing a problem which is learnable for any data distribution, and SGD is strictly better than RERM for any regularization function.
Descent-to-Delete: Gradient-Based Methods for Machine Unlearning
TLDR
This work gives the first data deletion algorithms that are able to handle an arbitrarily long sequence of adversarial updates while promising both per-deletion run-time and steady-state error that do not grow with the length of the update sequence.
Approximate Data Deletion from Machine Learning Models: Algorithms and Evaluations
TLDR
This work evaluates several approaches for approximate data deletion from trained models and proposes a new method with linear dependence on the feature dimension $d$, a significant gain over all existing methods which all have superlinear time dependent on the dimension.
Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks
TLDR
An upper-bound on the amount of information remaining in the weights is introduced, which can be estimated efficiently even for deep neural networks, and is a generalized and weaker form of Differential Privacy.
Mixed-Privacy Forgetting in Deep Networks
We show that the influence of a subset of the training samples can be removed – or "forgotten" – from the weights of a network trained on large-scale image classification tasks, and we provide strong
...
...