Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancement

Summary

The paper tackles the problem of machine unlearning, i.e. forgetting the influence of some data points, with remain-preserving manifold geometry. Authors dig deep into the gradient based approximate unlearning methods from the perspective of steepest descent and come up with a method beyond Euclidean constraints. To overcome computationally-intensive inverse Hessian calculation, authors proposed an efficient fast-slow weight update method to approximate the Hessian-adjusted direction. The theoretical findings are experimentally validated on CIFAR and TinyImagenet image classification problem and CIFAR and ImageNet image generation problems.

Strengths

1) The problem is very important to make current ML models more trustworthy and private. 2) Rigorous mathematical framework that unifies all the previous unlearning methods and introduce all the details of the proposed approach. 3) Interesting algorithm of overcoming inverse Hessian calculation by efficient fast-slow weight update. 4) Encouraging results and substantial improvements over previous baselines. 5) Easy-to-follow read with vivid illustrative figures.

Weaknesses

1) Only CV unlearning is presented. If your method works on ML algorithms in general, it would be nice to present some results on other tasks such as NLP, Audio, etc. 2) One interesting observation is that machine unlearning should make the model such that "forget" and "test" set results as close as possible, meaning FA and TA should be really close like in gold-standard RT model. However the proposed method performs not really the best in terms of this metric (|FA-TA|) compared to previous approaches on TinyImageNet. I genuinely don't know if it is a good metric, I might be wrong. Minor: 3. it would be nice to show with arrows ($\uparrow$ and $\downarrow$) in table 2 which metrics are the greater the better and which are the lesser the better.

Questions

1) How does your model remove the influence of one data point, when there's a request to delete one sample from the training dataset? This might be a practical scenario. 2) Are accuracy disparities actually good measure for unlearning comparison? It might be possible that two different models predicted right different samples, thus their accuracies are the same, while they are not close to each other. 3) Why does "R-only" models have greater RTE (longer time) compared to the "full SFR" model?

Rating

7

Confidence

2

Soundness

4

Presentation

3

Contribution

3

Limitations

Authors clearly defined the paper's broader impact and limitations in the appendix G.

Summary

This paper summarizes the previous gradient-based unlearning methods and proposes three essential components for unlearning: weighted forgetting gradient ascent, remaining gradient descent, and a weight saliency matrix. Then, this paper derives the steepest descent direction and proposes a fast-slow weight method to update the model parameters. The experiments in both image classification and image generation shows the effectiveness of the proposed method.

Strengths

1. This paper provides a new perspective to unify the gradient-based unlearning methods. 2. The experiments contain both image classification and image generation to show the effectiveness of the proposed methods 3. The derivation of the proposed method is attractive.

Weaknesses

1. The gradient-based unlearning methods usually make more changes to the latter layers than the previous layers, which leads to inefficient information removal. This paper does not consider the drawbacks of such unlearning methods. 2. For the image classification task, this paper only tries the ResNet18 model. Then, for the image generation, this paper only tries DDPM. It might not be enough to claim to ''form a unified MU approach for all CV unlearning tasks, including image classification and image generation.'' 3. This paper lacks more ablation studies to prove how SFR affect the unlearning results compared with other methods without the repairing stage on the updated model. 4. The main contributions of this paper are not clear. This paper claims to derive the steepest descent direction and proposes a fast-slow weight method. However, this paper still lacks a thorough advantage analysis of the steepest descent direction and the proposed update method.

Questions

1. How many remaining data does the proposed method require, and how many remaining data is actually used in experiments? 2. The calculation of Hessian matrix $(H_t^r)^{-1}$ still demands large computational resources. Does the proposed method have any approximation on it? 3. Although this paper claims the proposed methods can significantly reduce the optimization steps in unlearning, why does the RTE not show significant reductions compared with SCRUB and SalUn? 4. In the implementations, $\nabla\mathcal{L}^r(\theta_*)$, $\nabla C(\theta_t)$, and $\nabla R(\theta_t)$ may not be zero. Thus, how large are the influences of the three approximation operations in eqs 3,4,6.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

No further potential negative societal impact.

Summary

This work introduces a perspective to unify previous machine unlearning approaches by decomposing the gradient descent direction into three components including forgetting gradient ascent, remaining gradient descent, and weight saliency matrix. The steepest descent direction is derived on the remain-preserved manifold, and a fast-slow weight method is proposed to implicitly approximate online Hessian-modulated salient forgetting updates.

Strengths

1. This paper is well-written and well-structured. Each proposition is thoroughly proven in the appendix, and the preliminary section provides a detailed overview of the topic. 2. Experiments are comprehensive and encompass the most common settings in image classification and generation tasks. Additionally, the consideration of large pretrained models such as Stable Diffusion caters to practical scenarios.

Weaknesses

I am not familiar with the concept of machine unlearning. It seems like this work is rigorous and comprehensive. From my perspective, there are no obvious weaknesses.

Questions

Is the proposed method available for models in other modalities, apart from images? Including experiments on other modalities such as large language models could further broaden the scope of this work.

Rating

7

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

Please refer to the "Weaknesses" and "Questions" sections.

Thanks for your rebuttal.

Thanks for the authors' rebuttal. The rebuttal has solved my concerns on weaknesses 1, 3, and 4, and also on questions 2, 3, and 4. However, I still have the following three questions (not requiring extra experiments) 1. I acknowledge that currently, classification and generation are two major computer vision tasks for unlearning. Still, it does not mean all CV tasks consider the importance of image segmentation, object detection, etc. Although there are few works towards the unlearning on such tasks, they should also be considered if the paper wants to claim to work on all CV tasks. 2. Access to all the training data may not be practical, even in classification tasks. Then, I suggest that this paper can reduce the usage of training data or use a small public training data during unlearning, and it might also improve the time efficiency.