Automatic Clipping: Differentially Private Deep Learning Made Easier and Stronger

Per-example gradient clipping is a key algorithmic step that enables practical differential private (DP) training for deep learning models. The choice of clipping threshold R, however, is vital for achieving high accuracy under DP. We propose an easy-to-use replacement, called automatic clipping, that eliminates the need to tune R for any DP optimizers, including DP-SGD, DP-Adam, DP-LAMB and many others. The automatic variants are as private and computationally efficient as existing DP optimizers, but require no DP-specific hyperparameters and thus make DP training as amenable as the standard non-private training. We give a rigorous convergence analysis of automatic DP-SGD in the non-convex setting, showing that it can enjoy an asymptotic convergence rate that matches the standard SGD, under a symmetric gradient noise assumption of the per-sample gradients (commonly used in the non-DP literature). We demonstrate on various language and vision tasks that automatic clipping outperforms or matches the state-of-the-art, and can be easily employed with minimal changes to existing codebases.

Paper

Similar papers

Peer review

Reviewer 2WqJ7/10 · confidence 3/52023-07-05

Summary

The paper proposes a new alternative version of the DP-SGD algorithm where the gradients are normalized. This new method allows for a proof of convergence of the gradient when we add a small stability constant to the rescaling factor. This work eliminates the need for a costly 2D gridsearch between learning rate and clipping constant when training DP models. The authors then proceed to run experiments on to show their method equals or outperforms Abadi’s DP-SGD algorithm.

Strengths

The article eliminates a costly operation in DP training and provides strong evidence regarding the convergence of their method. The exhaustive list of datasets on which the method was tested testifies as to how only minimal changes to existing implementations are needed to apply this method. ### Originality The method is original. ### Quality The experiments presented are very complete and yield high quality results. Also, various supporting githubs and ressources are mentionned, making these results reproducible. ### Clarity The article is clear and well written. ### Significance The original motivation of the article is clear.

Weaknesses

## Sensitivity of the hyperparameter optimization The results are not clear enough to indicate whether the proposed DP training method is superior to the rescaled DP-SGD method which impedes the significance of the article. If the authors of the paper manage to prove that their method is less expensive than the rescaled DP-SGD trick then the contribution is significant. The article lacks a conclusive experiment or statement proving that the sensitivity of the hyperparameter optimization process is easier with this method than with the re-scaled version of Abadi’s DP-SGD. The paper shows in its appendix H. that the training process of the AUTO-S method is relatively robust to the choice of $\gamma$ on NLP tasks. However the paper doesn’t compare the sensitivity of the **AUTO-S method** to the $\gamma$ factor to the sensitivity of the **rescaled Abadi clipping** to the $R$ factor. In my opinion this is an important experiment to run, since it would justify using this method instead of the widely known rescaled DP-SGD version introduced in De et al. [15]. This could improve the impact and the significance of the contribution. ### Actionable feedback > However, R remains and is expensive to tune (l223) This sentence in particular is a bit of an overclaim in my opinion, see my comments below. **In your experiments you specifiy regarding the range of $\gamma$** > ”under which a wide range of 0.0001 < < 1 gives similar accuracy” and >“Note that the largest good is 1000 times bigger than the smallest good ” **In the comparison to related works on clipping** You state “Similarly (De et al.) re-parametrizes the per-sample gradient clipping to mimic normalization. However R remains and is expensive to tune (see figure 8 of De et al.)”. However this fig 8. shows that the re-parametrized per-sample gradient clipping seems to stay stable by a similar factor of approximately 1024 on the clipping norm, on a even harder dataset. Even though their experiment are not extensive enough to prove a superior efficiency of the rescaled DP-SGD method to yours, it is not conclusive enough for you to assume that your method is easier to tune. If you add this missing experiment, or if you modify this claim, I would consider improving my rating.

Questions

### Can you provide tangible proof that your method is significantly less expensive to tune than the rescaled DP-SGD algorithm ? The figure you cite in the concurrent work of De et al. in section 4.4 does not seem complete enough and gives no indication of the strain in ressources the rescaled DP-SGD might still induce.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The method is original. However some similarities can be drawn with existing concurrent approaches like Yang et al. 2022 [2206.13033] that did not go through peer-reviewing to the best of my knowledge, however this work might be worth citing.

Reviewer jhaz5/10 · confidence 4/52023-07-06

Summary

The author(s) proposed a new gradient clipping technique for differential private training algorithms. It was shown that the new clipping technique is more robust to hyper-parameters and can save the time for hyper-parameter tunning. Convergence of the proposed method under a gradient symmetric assumption is developed and experiments on both vision and language tasks are conducted to evaluate the proposed method.

Strengths

- The paper is well-written and easy to follow in general. - The experiments are thorough and the algorithm seems to be useful on the practical side.

Weaknesses

- The convergence analysis is based on a gradient symmetric assumption, therefore the utility bound in Theorem 4 is not really comparable to prior works. Please also see my quetions below. Overall, the theoretical contribution seems limited. However, given the strong empirical performance of the proposed method, this might be acceptable. - Some related work should be added [1,2,3]. [1] Clip21: Error Feedback for Gradient Clipping, https://arxiv.org/abs/2305.18929 [2] Improved Convergence of Differential Private SGD with Gradient Clipping, https://openreview.net/forum?id=FRLswckPXQ5 [3] Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimization, https://arxiv.org/pdf/2206.13033.pdf

Questions

- Consider the example with 3 data points, $f(x) = f_1(x) + f_2 (x) + f_3(x)$, where $f_1( x ) = 2 x^2, f_2(x) = (x-1)^2, f_3( x ) = f_2(x) = (x-1)^2$. Then the solution is at $x^* = 0.5$, but this point is not stationary for Algorithm 1, because $g_1( x^* ) = 2, g_2(x^*)=g_3(x^*)=-1$, which implies $g_1( x^* ) / \| g_1( x^* ) \| = 1, g_2(x^*) / g_2( x^* )=g_3(x^*)/g_3( x^* )=1$. Algorithm 1 willi fail on this example, does the author(s) agree?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I do not find any negative societal impact of this paper.

Reviewer FQ5E8/10 · confidence 5/52023-07-07

Summary

The paper discusses a way to control the sensitivity of DP-SGD by a method that does not require clipping.

Strengths

- The motivation for the paper is clear - The authors make a compeling argument that suggest why their method should work - They have sound theoretical guarantees and experiments - The idea is simple yet fully developed - The impact this could have on private learning is huge - Experimental results are compeling

Weaknesses

N/A

Questions

N/A

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

N/A

Reviewer V9NQ4/10 · confidence 4/52023-07-14

Summary

This paper introduces 'automatic clipping' to replace the usual clipping operation in DP-SGD. The challenge of usual clipping is to choose a good threshold 'R' especially for deep learning models. The authors claim that automatic clipping maintains the same level of privacy and computational efficiency as existing DP optimizers such as DP-SGD, DP-Adam, and DP-LAMB, but without the need for any DP-specific hyperparameters. In fact, automatic clipping uses a kind of normalization to bound the sensitivity of individual gradient. Additionally, they provide a thorough convergence analysis for DP-SGD with automatic clipping in the non-convex setting, demonstrating that it can match the convergence rate of standard SGD under certain conditions. The authors validate their proposal by showing that automatic clipping either matches or surpasses the state-of-the-art performance on a variety of language and vision tasks.

Strengths

1. **Simplicity**: The proposed method of automatic clipping significantly simplifies the process of DP training by eliminating the need for tuning DP-specific hyperparameters, thereby making DP as user-friendly as standard non-private training. 2. **Performance**: The paper demonstrates that the automatic clipping method is either on par with, or better than, existing state-of-the-art techniques in a variety of tasks. This suggests that the method does not compromise performance for ease-of-use. 3. **Rigorous Analysis**: The authors provide a detailed convergence analysis of automatic DP-SGD in a non-convex setting, showing that it can match the convergence rate of standard SGD under the symmetric gradient noise assumption of per-sample gradients.

Weaknesses

**Misleading justification**: In figure 2, the paper present the dot products of $\langle g_i, \sum_i g_i \rangle$. First it does not make sense if the dot product is properly normalized. Moreover, the distribution of $\langle g_i, \sum_i g_i \rangle$ does not necessarily align with good performance. The illustration in Figure 2 is rather misleading. **Too strong Assumptions**: The proposed automatic clipping method's performance is based on a symmetric gradient noise assumption of the per-sample gradients. This assumption is too constrained and does not satisfy in practice. **Overclaim**: Section title 4.3 "Automatic clipping is equally private and maximizes utility" is somewhat overclaiming. It is not carefully argued why automatic clipping can "maximize utility given a privacy budget. **Missing reference**: The paper missed a very related reference Yang et al. "Normalized/clipped sgd with perturbation for differentially private non-convex optimization" which also studied the DP-SGD with normalization. The paper should have a careful comparison with it.

Questions

See weakness.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

See weakness

Authorsrebuttal2023-08-13

Follow up on symmetric gradient noise assumption

We would like to provide more references to justify that our assumption follows from the non-DP deep learning literature (hence it is actually less constrained). Most theoretical papers that analyze the standard SGD assumes that the mini-batch gradient follows $$\frac{1}{B}\sum_i g_i\sim \frac{\partial \hat L}{\partial w}+\xi(w)$$ where $\frac{\partial \hat L}{\partial w}$ is the oracle gradient with respect to the generalization loss, and $\xi$ is the random gradient noise with $\mathbb{E}[\xi]=0, \mathbb{E}[\xi\xi^\top]=\Sigma(w)/B$. That is, the per-sample gradient (by setting $B=1) follows $$g_i\sim \frac{\partial \hat L}{\partial w}+\xi(w)$$ Next, the noise structure $\Sigma$ is proportional to the oracle Hessian matrix $H$, which is symmetric: - [2*,3*,4*] below assume $\Sigma(w)=\sigma^2 H(w)$ for some constant $\sigma$ - [5*,6*] below assume $\Sigma(w)=2L(w) H(w)$ for loss $L$ - [46, 63, 12, 70] in our submission assume $\Sigma$ is the covariance matrix of some Gaussian. **In all these cases, the gradient noise is symmetric**, which is verified in practice as well (happy to add reference for this as well). We hope this is helpful and would appreciate it if the reviewer could increase the score. [2*] Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In International Conference on Machine Learning [3*] Stanisław Jastrz˛ebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in SGD. arXiv preprint arXiv:1711.04623, 2017. 3 [4*] Zhiyuan Li, Tianhao Wang, and Sanjeev Arora. What happens after SGD reaches zero loss? –a mathematical framework. In International Conference on Learning Representations, 2022. 3 [5*] Liu Ziyin, Kangqiao Liu, Takashi Mori, and Masahito Ueda. Strength of minibatch noise in SGD. In International Conference on Learning Representations, 2022. 3 [6*] Takashi Mori, Liu Ziyin, Kangqiao Liu, and Masahito Ueda. Logarithmic landscape and power-law escape rate of SGD. arXiv preprint arXiv:2105.09557, 2021. 1, 3, 8

Authorsrebuttal2023-08-18

Further discussion

We would like to thank the reviewer again for the comments and hopefully our response clears your concerns. Given that the discussion period ends in 3 days, we are happy to address further questions, and if there are no more concerns, we would appreciate it if the reviewer can consider raising the score.

Authorsrebuttal2023-08-20

We hope we have addressed your comments and if not, we are happy to further discuss them. Given that the discussion period ends in 1 day, we would appreciate it if the reviewer can consider raising the score.

Authorsrebuttal2023-08-21

Dear reviewer, Thank you again for the discussion. We hope we have addressed your concerns about symmetric noise of gradient and tightness of upper bound. Given that the discussion will end in an hour, please let us know any questions you may have. It would be really appreciated if our response is reflected in your score.

Reviewer Cysx7/10 · confidence 4/52023-07-25

Summary

The script proposed an automatic clipping methods for various DP algorithms. The problem is important because the performance of DP models are sensitive to the choice clipping threshold, yet there is no theorical guidance for tuning it.

Strengths

The writing is clear and the paper is easy to follow. The idea is simple and effective. Theory and experiments are provided to show the efficacy of the proposed automatic clipping method. I think this result is worth sharing with the DP community.

Weaknesses

See below

Questions

**Only one question:** According to to line 140, Auto-V maximizes the dot product similartiy **only when** <g_i, \sum_j g_j> >0 for any i. However, in practice, <g_i, \sum_j g_j> could be negative, but Auto-V still perform gradient normalization on these samples. I wonder: what if we only normalize g_i with <g_i, \sum_j g_j> >0 and ignore the rest, (i.e., do exactly the optimal normlization in line 140). Will it bring better performance? Similar question also goes to Auto-S. **Suggestions:** I suggest add index to the equations in line 138 and 140.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See above

Reviewer Cysx2023-08-12

Thanks for the response

Thanks for the response. I will keep my score

Reviewer jhaz2023-08-15

Thanks for your rebuttal

I would like to thank the author(s) for the detailed rebuttal. I think the example raised in my question is still a validation toy example of the empirical-risk minimization problem (finite-sum); it can be sampled from a mixture of Gaussian's distribution. Anyway, I still appreciate the empirical performance of the proposed method and keeping my score unchanged.

Reviewer V9NQ2023-08-15

thanks for the response. Explanation is still misleading.

Dear authors, Thanks for the response. I found the explanation is still misleading because the upper bound $-\eta \langle \sum_i g_i, \sum_i C_i g_i + \sigma N(0,1)\rangle + \frac{L}{2}||\sum_i C_i g_i + \sigma N(0,1)||^2$ is not carefully analyzed. You claim that "The first term gives the term you want to maximize by ignoring the noise gradient inner product, and the second term is independent of clipping" is not correct. It is obvious that the second term depends on whether you are using the clipping or normalization. One can consider a simple set-up: $g_i$ is extremely small, $C_i$ for normalization will be much larger than 1 (clipping threshold). That means when you doing normalization, although the first term is maximized, the second term may also be (unwantedly) magnified significantly. Overall, it is not clear whether the upper bound is increased or decreased.

Authorsrebuttal2023-08-15

Clarification on the second term

Thank you for the discussion! We would like to clarify why we may treat the second term as independent of whether we use clipping or normalization. Let's consider the re-parameterized clipping as in [16] De et al.: $C_i=\min(1/||g_i||,1/R)$ and AUTO-V: $C_i=1/||g_i||$ (the analysis will be the same if we consider Abadi's clipping $C_i=\min(R/||g_i||,1)$ and R-dependent AUTO-V $C_i=R/||g_i||$). In our proof above Equation (C.2), with $Z=N(0,I)$, we used $$\frac{1}{2}||\sum_i C_i g_i+\sigma Z||^2\leq ||\sum_i C_i g_i||^2+\sigma^2||Z||^2\leq B\sum_i ||C_i g_i||^2+\sigma^2||Z||^2$$ given by Cauchy Schwartz inequality. Then in line 608, it is clear that $||C_i g_i||\leq 1$ for both clipping and normalization and everything (related to the second term) follows since after Equation (C.2). Therefore, **the proof as a upper bound guarantee is valid (though maybe not tight)**. To justify the "tightness", we only need to justify the use of $||C_i g_i||\leq 1$ for clipping, which can be much loose if most $g_i$'s are extremely small and not clipped, as you correctly pointed out. Hence the question is what proportion of $g_i$'s is clipped. Theoretically, if R is small, then all $g_i$'s are clipped and $C_i=\min(1/||g_i||,1/R)=1/||g_i||$ reduces to AUTO-V. In fact, this scenario has been observed to achieve strong performance by recent works (see [16] Figure 8 and [40] Figure 8(b)). Empirically, from **Figure 12 (appendix H)** and line 799, we observe that $30\sim 100$% per-sample gradients are clipped, across different datasets. Specifically, for GPT2 training, *100% of $g_i$ are clipped in all iterations*. We agree this proportion is certainly task-dependent, but it should provide helpful insights to view the second term as independent of clipping or normalization.

Reviewer 2WqJ2023-08-18

Thank you for this detailed response. I hope to see this discussion on De et al. in the camera-ready version. I have a better understanding of your contribution. I increased my score to 7 as a consequence.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC