State-of-the-art approaches for training Differentially Private (DP) Deep Neural Networks (DNN) face difficulties to estimate tight bounds on the sensitivity of the network's layers, and instead rely on a process of per-sample gradient clipping. This clipping process not only biases the direction of gradients but also proves costly both in memory consumption and in computation. To provide sensitivity bounds and bypass the drawbacks of the clipping process, we propose to rely on Lipschitz constrained networks. Our theoretical analysis reveals an unexplored link between the Lipschitz constant with respect to their input and the one with respect to their parameters. By bounding the Lipschitz constant of each layer with respect to its parameters, we prove that we can train these networks with privacy guarantees. Our analysis not only allows the computation of the aforementioned sensitivities at scale, but also provides guidance on how to maximize the gradient-to-noise ratio for fixed privacy guarantees. The code has been released as a Python package available at https://github.com/Algue-Rythme/lip-dp
Paper
Similar papers
Peer review
Summary
- The paper investigates the use of Lipschitz constrained networks to replace clipping functions and limit gradient sensitivity in DP-SGD. - Lipschitz constrained networks are utilized as an alternative to clipping in order to address the issues of clipping's impact on convergence and performance in DP-SGD.
Strengths
- The idea of removing clipping as an alternative to clipping itself is promising, as clipping is known to have detrimental effects on convergence and performance of DP-SGD, even without noise addition [1]. - The paper introduces the replacement of Vector-Jacobian product with Scalar-Scalar product to reduce computational complexity. The proposed methods outperform existing SGD approaches by a significant margin in terms of speed, which is crucial as memory usage and time inefficiency are major drawbacks of DP-SGD. [1] Differntially Private Shaprness-Aware Training (ICML’23)
Weaknesses
- Please refer to the questions. - (Minor) There are several typos, such as the use of "cotangeant vector" which sounds little awkward, and inconsistencies in figure references (e.g., Fig 4 vs. Figure 5). Please carefully review the grammar and correct the typos.
Questions
- The main concern is whether "Clipless DP-SGD" is truly effective compared to "Clip DP-SGD" in terms of training stability and performance. While it is understood that the main idea is to address the detrimental effects of clipping by proposing a clipping-free approach, the proposed method still relies on approximations or has drawbacks such as the Lipschitz constrained condition or gradient approximation with scalar-scalar form. Thus, (i) Can you provide mathematical bounds to support the strength of your methods? or (ii) Can you provide additional experimental results beyond Figure 5 to further demonstrate the performance of your approach? - Recent papers have explored different ways of using DP-SGD, particularly in fine-tuning tasks, to mitigate memory usage and time inefficiency [2,3]. Considering that their approaches may differ from training from scratch tasks, it would be worth to investigate whether your proposed networks are well-suited for fine-tuning tasks to improve generalization performance. - Can you provide more details on the implementation of GNP networks in DP-SGD? The implementation of 1-Lipschitz seems to be naive and detrimental to traininig, similar to selecting a norm clipping value in clip DP-SGD. However, the authors mention a problem in this implementation, stating, "With Gradient Norm Preserving (GNP) networks, we expect to mitigate this issue" (line number 236). Could you further explain how your methods address this problem? or let me know that my understanding is incorrect. I’d be happy to raise my score if the authors can address the weaknesses/questions in the rebuttal. [2] Large language models can be strong differentially private learners (ICLR’22) [3] Scalable and efficient training of large convolutional neural networks with differential privacy (NeurIPS’22)
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
4 excellent
Limitations
The paper provides a detailed discussion of its limitations.
Summary
Differentially Private (DP) Deep Neural Networks (DNNs) face challenges in estimating tight bounds on the sensitivity of the network’s layers. Instead, they rely on a per-sample gradient clipping process (as argued by the authors). This process not only biases the direction of the gradients but also proves costly in both memory consumption and computation. To provide sensitivity bounds and avoid the drawbacks of the clipping process, the authors provide a theoretical analysis of Lipschitz constrained networks, and uncovers a previously unexplored link between the Lipschitz constant with respect to their input and the one with respect to their parameters. By bounding the Lipschitz constant of each layer with respect to its parameters, the authors argue it will guarantee DP training of these networks.
Strengths
The paper is well-structured and clearly written. The theoretical part is simple and easy to follow.
Weaknesses
Estimating Lipschitzness with respect to parameters may not be necessary. If the network is Lipschitz continuous with respect to the input, its gradient will be bounded, and thus the weight update will also be bounded. So, the motivation may not be rational. Experimental results do not support the arguments. The validation accuracy of the DP-SGD is lower than several referenced works.
Questions
1. Adam uses a normalized gradient between the first-order momentum and square root of the second-order momentum, which makes it robust to Lipschitz variations in different layers. Does it still need gradient clipping? 2. Regarding per-sample gradient clipping, in the current deep learning framework, we usually do not use per-sample gradient clipping. Instead, we apply gradient clipping after obtaining the averaged gradients. In the abstract, the authors mention the difficulty of per-gradient clipping, but I don't understand the reason? 3. Also, gradient clipping is a fast operation. I don't understand why it results in higher memory consumption and computation. 4. Regarding the experiments, the results do not show that the DP-SGD achieves better performance (validation accuracy) than the referenced works. How should we evaluate the effectiveness of this work?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
See weaknesses and my questions
Summary
The paper addresses the problem of efficiently bounding the sensitivity of gradients in DP-SGD by using special architectures the layers of which can be proven to be Lipschitz with respect to the parameters, hence bounded gradient. They show how to recursively calculate the sensitivity of a sequence of layers, and incorporate the method in an algorithm to perform private SGD without clipping.
Strengths
The writing is exceptionally lucid. The concept is original and potentially significant, although there are at present many limitations.
Weaknesses
There are a lot of constraints on the architecture that severely limit the potential of the method for short-term impact. I'm torn, because introducing the concept at this stage is of value, but far more work must be done -- both theoretical, in establishing the requisite bounds for popular architectures -- and experimental, in demonstrating that the approach achieves good points on the privacy/utility/efficiency Pareto frontier -- before we can assess the significance of the work. I'm not convinced that it isn't a major problem that the gradients can vanish during training. This is the reason for the success of adaptive (layer-wise) clipping strategies. In particular see "EXPLORING THE LIMITS OF DIFFERENTIALLY PRIVATE DEEP LEARNING WITH GROUP-WISE CLIPPING" which would seem to enjoy the efficiency of your approach without the drawbacks of vanishing gradients or restricted architecture class.
Questions
What is meant by “the activation is centered in zero”-- in particular, is ReLU really centered? Experimental results. Figure 4 is too small. I can't see tensorflow_privacy? Also it appears that the advantage over optax is negligible. In Figure 5, the other algorithms get one or a few points, while their own algorithm is given the benefit of thousands of hyperparameter choices to establish the Pareto frontier. Moreover the result is negative: for EMNIST the other algorithms match yours, and for the other datasets the other algorithms have higher accuracy. Probably the DP definition should be using add/remove-one neighborhoods, in accordance with the use of (pretend) Poisson sampling for accounting.
Rating
3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
4 excellent
Contribution
2 fair
Limitations
Limitations are honestly and adequately discussed. No potential negative societal impact.
Summary
The paper studies the question of how to do differentially private optimization without using per-sample gradient clipping, in order to simplify and speedup the iteration cost. The paper proposes to restrict the class of functions to feed-forward neural networks for which it is feasible to compute bound on the gradient norm (Lipshitz constant), and proposes to compute adaptively the bound on the gradient norm (layer-wise) at every step of DP-SGD depending on the current iterate point. Paper provides the description of the algorithm, as well as evaluates its practical behavior.
Strengths
- An efficient implementation of the algorithm is provided. - Experiments show that per-iteration runtime of the proposed algorithm is indeed faster. - Overall the paper is interesting and novel and provides a new direction for future research.
Weaknesses
1. No clear comparison of the proposed algorithm to the baseline method (DP-SGD) is given in terms of the final accuracy. When restricting to the same architecture, it is unclear if the proposed algorithm can still reach the good accuracy compared to the classical DP-SGD with gradient clipping. Without clipping the gradients, the amount of the added DP noise to each gradient is larger than if you clip the gradients, which might hurt the final performance. 2. From the experiments on CIFAR10 one might conclude that for the same privacy $\epsilon$ the final accuracy of the baselines is much better than of the proposed algorithm, which makes the proposed algorithm not applicable. 3. In the “local” strategy (line 201), how exactly did you calculate the amount of the noise to be added? I did not find a clear description of the “local” strategy, and how it is different from the “global” strategy. 4. Some parts of the paper are not very clearly written (see questions below).
Questions
1. What do the green lines in pareto-front on Figure 5 represent ? 2. Remark 1 says that the paper proposes a more efficient way to compute X_d, however I did not find the algorithm in the paper. 3. It is unclear from the current presentation what is the connection of Theorem 1 with Algorithm 1, is it only for the easier computing of step 6 of Algorithm 1 ? What about steps 2 and 7? 4. I did not understand the paragraph in lines 258-267. However some hybrid approach might be indeed a good solution: if the estimated worst case gradient norm is large, the training might benefit from clipping in order to reduce the amount of noise added to the gradients, however, when estimated worst case gradient norm is small, the training would benefit of using the estimates and have little added noise. 5. Many experimental details are missing: which architecture/hyperparameters did you use, etc. 6. A very minor comment: when printing on the paper, formulas in green and yellow color are not very visible, try using a different color.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
yes
Response to rebuttal
I appreciate that it should not be necessary to beat all SOTA results as a precondition for initial publication. But I feel there is a high potential for this method to produce poor models in terms of accuracy, and there are not sufficient experimental results to demonstrate that this is not occurring. My concern is the same as reviewer 6vik's weakness #1. Without clipping, if you have very small gradient norms, your signal-to-noise ratio is low, and the trained model will have poor performance. Clipping lets you control this signal-to-noise ratio more effectively, particularly wtih adaptive clipping, or better yet layer-wise adaptive clipping, which also solves the efficiency problems around per-example clipping. So I feel there needs to be more clear empirical evidence that the method does not suffer compared to those methods in terms of accuracy. Another idea that might help to alleviate this concern would be to actually measure the empirical gradients and report their distribution: what is the empirical signal-to-noise ratio? But that would be outside of the scope of this review process. I maintain that it is not fair to show your Pareto frontier compared to a single point from another algorithm. I see your point that ideally we would all be publishing our Pareto frontiers, but since we are not, the honest comparison is to run your own hyperparameter tuning and to compare it to the other published result. Thank you for clarifying that the architectural constraints are not as severe as I had thought. I will increase my score to 3 and decrease my confidence.
Thank you for opening this discussion. > Another idea that might help to alleviate this concern would be to actually measure the empirical gradients and report their distribution We did it in **figures 6.b) and 7.b) in appendix p17**. We see that the gradient norms for GNP networks is *higher* than the one of conventional networks. > Without clipping, if you have very small gradient norms, your signal-to-noise ratio is low, and the trained model will have poor performance. The signal-to-noise ratio is not the only metric that influences the final utility: the gradient need not to be unbiased for the signal to be actually useful. By getting rid of clipping, we also get rid of the induced bias. > I appreciate that it should not be necessary to beat all SOTA results as a precondition for initial publication [...] I see your point that ideally we would all be publishing our Pareto frontiers We agree; that is indeed a fairer experimental procedure, and our work pushes towards this ideal. > I feel there is a high potential for this method to produce poor models in terms of accuracy [...] Note that in the rebuttal we provided 2 new datasets on which the final utility is competitive (or exceed) the one of vanilla DP-SGD. > I maintain that it is not fair to show your Pareto frontier compared to a single point from another algorithm [...] the honest comparison is to run your own hyperparameter tuning and to compare it to the other published result When offering a library, average behavior becomes more indicative of typical performance than seeking out the extremes. In a scenario with an unlimited computational budget, even a random baseline holds a nonzero chance of outperforming the current state of the art (SOTA). Hence, when we emphasize the highest score in our reporting, what are we truly measuring apart from the computational budget? However, we understand your overall concern, please see this additional experiment of Clipless DP-SGD that uses standardized Cifar-10 dataset (assuming no privacy loss in computing mean and std, just like Opacus tutorial): | **Epsilon** | 4.05 | 5.0 | 8.01 | 11.1 | 15.03 | | ----- | ------ | ------ | ------ | ----- | ---- | | **Val. Acc.**| 37.6 | 39.3 | 42.0 | 43.2 | 44.5 | We see that we close the gap with the results of De et al, which are among the top-performers on Cifar-10.
Thank you for pointing me to the gradient norm distribution figures in the appendix. In figures 6/7.b) what are the parameters for which this gradient norm distribution is computed? Random or trained? It is stated that the norms are "sticking closer to the upper bound we are able to compute for the gradient norm" -- I can see that the gradients are larger for the GNP network, but what is that norm bound? Sorry, I cannot find the work you are comparing to with these new experiments, De et al. Can you point me to the exact table of values I should compare your accuracies with?
Thank you for your engagement in our work. We appreciate your thoroughness. In Fig 6. and 7. the report the bounds for Dense layers in a MLP of depth 3 > Random or trained? It is computed on the whole dataset at initialization. > In figures 6/7.b) what are the parameters for which this gradient norm distribution is computed? In fig 6., we agregate the individual bounds on the layers to obtain a global bound on the whole gradient vector, whereas in fig 7. we plot the distributions of individuals layers. > but what is that norm bound? In fig 6.b), the global upper bound is 135.66. Less than two times the maximum empirical bound of 73. In fig 7.b) the common upper bound of each of the gradient bound (for every layer) is 78.38. Not too far away from the maximum empirical bound of 56. For the conventional network without Lipschitz constraints we don't have analytical upper bounds. We will add these informations to the figure. > Sorry, I cannot find the work you are comparing to with these new experiments, De et al. Can you point me to the exact table of values I should compare your accuracies with? The closest one is the first row of their table 2 (page 7) where their model is a Wide-Resnet, and where they rely on the same pre-processing.
So fig 7b shows the distribution of the gradient norms of a GNP at initialization (randomly initialized parameters). What about at other points in training? Would you expect the gradient norm distribution to shift toward zero as the model converges? For a more clear picture, we would need to see this gradient norm distribution during the main phase of learning when losses are dropping, and again at convergence. Thank you for sharing the upper bounds, but to be honest I am disappointed that I had to ask for them: since you referred to it ("sticking close to the upper bound") it is misleading that you didn't already write it in the paper so the reader can judge for themselves whether it is close or not. Now you want to compare the maximum gradient norm at initialization to the norm bound to argue you have a good signal-to-noise ratio. But the mean/median/mode looks more like 20-30 in 7b, or about 3-4x smaller than the bound. I appreciate your point that bias from clipping is also worth considering, but having noise on a scale of 3-4x the typical gradient will surely degrade performance significantly. Now you claim to "close the gap" to De et al., where their most basic model with eps=8 has a validation accuracy of 50.8 compared to your least private model (eps=15) with 44.5. I'm sorry, I actually really want to support this work: as I said the writing is very clear, the idea is interesting, the experiments are extensive and it's wonderful that you published the library, but in the end I am simply not convinced that the method will actually be useful to anyone in practice due to this issue of having to add so much noise relative to the gradient norms. Clipping may have its downsides, but it is simple and relatively effective, particularly with (layer-wise) adaptive clipping.
> I actually really want to support this work: as I said the writing is very clear, the idea is interesting, the experiments are extensive and it's wonderful that you published the library Thank you for your encouraging words. Increasing your rating is the best support for our work you can provide here. We understand your motivations for rejection, but we disagree on the substance. The peer-review process is integral to refining and validating our approach. We are encouraged by the positive responses from all reviewers, including yourself, regarding the validity of our method. The 'lipdp' package serves multiple purposes within our work. It not only facilitates reproducibility but also extends a valuable resource to the community. By offering 'lipdp,' we aim to streamline the evaluation of enhancements built upon "vanilla" Clipless-DP-SGD. It is important to note that Clipless-DP-SGD is a standalone contribution, deserving its own dedicated focus. Integrating other methods within the same paper might compromise clarity and lead to potential misinterpretations of the true contribution. > Would you expect the gradient norm distribution to shift toward zero as the model converges Indeed, the gradient of the loss w.r.t the logits $\nabla_{y}\mathcal{L}$ tends toward zero when the accuracy is high: reaching high accuracy is exactly what we want. And if the accuracy is not high, then the gradient norm remains high too. This quantity is a common upper-bound of all other gradient bounds. The same phenomenon occurs in vanilla DP-SGD: at some point in training the noise overrun the signal when gradients are smaller than the clipping value. We will add the corresponding figure showcasing the evolution of the histograms. > I appreciate your point that bias from clipping is also worth considering, but having noise on a scale of 3-4x the typical gradient will surely degrade performance significantly. You raise a valid concern, but you are also missing an advantage of our method: by getting rid of per-sample gradient computations, we allow higher batch size more easily. We managed a physical batch size of *50,000 examples* in figure 4. Only Opacus managed similar performance, *with virtual batching*. This higher batch size increases the signal-to-noise ratio in proportion. > it is misleading that you didn't already write it in the paper We agree it should be part of the figure, we modified the paper in consequence.
So, toward convergence, the gradient norms become small, while your norm bounds remain constant, decreasing signal-to-noise, is that right? Yes, the same happens with vanilla DP-SGD, but with adaptive clipping, the clipping norm decreases in tandem with the gradient norms, so the signal-to-noise is approximately constant.
> but with adaptive clipping, the clipping norm decreases in tandem with the gradient norms, so the signal-to-noise is approximately constant Adaptive clipping consumes a part of the privacy budget to compute adapt the clipping value, and it introduces new hyper-parameters to the method. Arguably, it is more complex: "*One possibility is to use adaptive clipping instead of fixing the clipping norm a priori Andrew et al. (2021). However, its implementation is more complicated than static clipping, and thorough tuning of hyperparameters with static clipping norm usually results in the same utility as adaptive clipping Andrew et al. (2021).*" **Source**: Ponomareva, N., Hazimeh, H., Kurakin, A., Xu, Z., Denison, C., McMahan, H.B., Vassilvitskii, S., Chien, S. and Thakurta, A.G., 2023. *How to dp-fy ml: A practical guide to machine learning with differential privacy.* Journal of Artificial Intelligence Research, 77, pp.1113-1201. Therefore, we don't think that adaptive clipping is a sufficient argument on itself to rule-out the relevance of our work. Finally, note that our framework **is compatible with adaptive clipping**: the clipping layer that clips $\nabla_y\mathcal{L}$ can benefit from this innovation. We did experiments in this direction, and it improved our final accuracy. However, this is outside the scope of Clipless DP-SGD framework to present this, and is more appropriate as a future work.
The authors are simply not offering credible arguments. Adaptive clipping consumes around 0.5% of the noise budget, and works well out of the box without hyperparameter tuning [1]. It may be that static clipping performs about as well, but as Ponomareva et al. say, "thorough tuning of hyperparameters" is necessary. Actually, in the original adaptive clipping paper [1], adaptive clipping performs better than fixed clipping in three out of six datasets even after tuning the fixed clipping norm, so the DP-ify paper may have mischaracterized it. I will raise my confidence back to 4. [1] Andrew et al. (2021) "Differentially Private Learning with Adaptive Clipping"
I would like to thank the authors for their clarifications that addressed most of my concerns, as well as for extra experiments of DP-SGD. My only remaining question is: Remark 1 & 2. Would it be possible to fix architecture to Lipschitz networks and compare to DP-SGD on these networks? As well as to fix data preprocessing, and compare to the algorithms on the exactly same fixed setting (same data preprocessing & same architecture) ?
Thank you for your continuing effort in engaging the discussion. > Would it be possible to fix architecture to Lipschitz networks and compare to DP-SGD on these networks? Could you clarify your question? From what we understand, you are curious about the performance of DP-SGD on GNP networks? If so, take a look at figure 11 in appendix D3, p26. > on the exactly same fixed setting (same data preprocessing & same architecture) ? There isn't an exact 1-to-1 mapping between GNP and conventional networks. For example, residuals connections are argued to mitigate the vanishing gradient phenomenon, but this is redundant with the orthogonality condition of GNP networks. Other difficulties exist: the manifold dimension of orthogonal matrices is half the one of all matrices, so at equal width the number of degrees of freedom is actually smaller. We attempt a run of Clipless DP-SGD on Cifar-10 with a MLP Mixer, with a standardized Cifar-10 (assuming no privacy cost in computing mean and std, just like in optax tutorial) and we obain: | **Epsilon.**| 4.05 | 5.0 | 8.01 | 11.1 | 15.03 | | ----- | ------ | ------ | ------ | ----- | ---- | | **Val. Acc.**| 37.6 | 39.3 | 42.0 | 43.2 | 44.5 |
Thank you very much for your reply. I meant comparison the performance of DP-SGD with clipping vs Clippless DP-SGD on the same plot for the same GNP networks.
> I meant comparison the performance of DP-SGD with clipping vs Clippless DP-SGD on the same plot for the same GNP networks. Indeed, we can merge figure 5 and figure 11. The same architectures have been tested in fig 5 and fig 11. Empirically, the Pareto front is about the same, but the experiments of GNP+Clipping (fig. 11) were much slower to run than GNP+Clipless (fig. 5).
I would like to thank the authors for their replies. I have raised my score by 1 point.
Thank you for responses and clarifications that the authors provided. - One of my remaining questions is about the performance aspect. As my first question is little ambiguous, I point out that my original question point was to highlight that GNP appears to be inherently more effective than gradient clipping in the context of deep learning optimization even though theoretically the proposed methods have smaller upper bound. My concern arose after observing the CIFAR-10 accuracies depicted in Figure 5. Although the authors mention the potential for future integration, it seems, in my view, that employing GNP is still worried rather than using gradient clipping. At least in my opinion, using GNP is better than gradient clipping when $\epsilon$ is small.
Thank you for your insighful question. > At least in my opinion, using GNP is better than gradient clipping when is small. Your remark raises interesting questions about the dynamics of DP training in neural networks. This reminds us the following excerpt of De et al (p6 of their work) that we’d like to share here: "*[...] reducing the variance introduced by noise may be more important than reducing the bias introduced by clipping.*" We remind their concern was attaining high accuracy in higher $\epsilon$ regimes. Gradient clippings reduces variance at the cost of increased bias. Clipless DP-SGD removes the bias but suffers from higher variance if the gradient bounds are too lose. If your hypothese is correct, it means that low bias is more important is low $\epsilon$ regimes, and reduced is more important is higher $\epsilon$ regimes. Intuitively, that seems natural. We tested your hypothesis by tuning hyper-parameters in the small $\epsilon$ regime, and we obtain the following results on Cifar-10: | **Epsilon** | 3.5 | 4.1 | 4.9 | | ---- | ----- | ---- | ----- | | **Val. Acc.** | 38.9 | 40.4 | 42.1 | With smaller epsilon constraints: | **Epsilon** | 0.75 | 0.96 | 1.14 | ---- | ----- | ---- | --- | | **Val. Acc.** | 31.5 | 34.2 | 35.2
Official Comment by Reviewer Coyf
Thank you for your responses and clarifications. - I remain unconvinced by the argument that per-sample gradient clipping significantly enhances speed. Consider the computational complexity of a convolution operator, which stands at $O(C_{in} \times C_{out} \times K \times K \times W \times H)$. In contrast, gradient clipping necessitates only O($C_{in} \times C_{out} \times K \times K$) comparison operators. Similarly, for a linear layer, the computational complexity is $O(C_{in} \times C_{out} \times D)$, while gradient clipping requires only O($C_{in} \times C_{out}$) comparison operators. In both convolutional and linear layers, the reduction in computation appears negligible. - Numerous studies delve into the realm of local or global Lipschitz continuity. I remain skeptical about the challenge in guaranteeing bounded inputs, loss, and bias terms. Thus, I prefer to keep my previous rating.
> I remain unconvinced by the argument that per-sample gradient clipping significantly enhances speed I think there is a misunderstanding here: **per-sample gradient clipping is the solution of litterature, not the one of our paper**. Beside, as we explain in our review, the computional cost comes from per-sample gradient computation. We are confused by your review. It feels like your criticizing the whole field of research rather than our contribution. You say you remain unconvinced, but did you read the references we sent ? If you believe that speed is not an issue in DP training, are you suggesting that existing libraries like Opacus or tf_privacy are not implemented correctly ? Are you suggesting that the survey "*How to dp-fy ml*" (that was presented in ICML 2023 as a tutorial) is ill-informed on the challenges of the field ? How do you explain the speed improvement of our framework over concurrent frameworks if you think the problem does not exist at all ? > Similarly, for a linear layer, the computational complexity is [...] convolutional and linear layers, the reduction in computation appears negligible. You omitted to take into account the batch size $B$ in your complexity evaluation. Your computations are only valid for gradient clipping on the *average gradient*, whereas the field of DP training relies on *per-sample gradient* computations. Therefore, it is not relevant for DP training. We suggest you take a look at Section 4 of the seminal paper of Abadi et al. We hope their analysis can convince you. > Numerous studies delve into the realm of local or global Lipschitz continuity. I remain skeptical about the challenge in guaranteeing bounded inputs, loss, and bias terms. Would you mind citing a *single one* of those "numerous studies" that provides formal Lipschitz guarantees over parameter space like we do ? Can you show us a *single neural network* (beside ours) for which you know the global upper bound of the Lipschitz constant over the parameter space ?
Decision
Reject