Pre-training Differentially Private Models with Limited Public Data

Summary

This work theoretically justifies the loss of utility under DP pre-training under the lens of a Hessian. This theoretical result can then be leveraged to perform efficient pre-training of models on public data (small amount) to significantly improve DP-trained model's performance.

Strengths

The problem authors attempt to solve in this paper is very important and the work is timely (given the concerns re:usage of copyrighted material during pre-training of large-scale models). To the best of my knowledge, the results are theoretically justified and sound. There are many experiments supporting authors' claims in this work, outperforming the existing approaches.

Weaknesses

My fundamental issue with the positioning of this work is that we already know that public data improves DP - [1,2,3,4,6]. We also know that DP training during pre-training is harming utility more than it would have harmed the pre-trained model [2,3,4]. Therefore my question is what is the main scientific output of this paper? The conclusions are already widely established and I do not see any new outcomes that can be derived from the conclusions of this work. It would have been interesting to see if this result it linked to model's generalization/memorization capacities as described in [5], showing some novel contextualized understanding of the training dynamics, but there was no discussion on the interplay of the training time and memorization capacity under DP. The theoretical justification of why this phenomenon arises is indeed interesting and can be useful, but on its own does not provide any novel insights into the training process that are 'actionable'. This is, again, because it was already previously established that a) DP pre-training harms utility [1,6,7], b) public data aids utility of DP models [2] and c) when models are untrained, they are more susceptible to various phenomena that can affect the training dynamics (e.g. loss of information via clipping [5]). So I am struggling to understand what can one gain from the results of this work? The 10% data threshold is a purely empirical result, which may or may not scale to other settings and datasets. [1] - Bu, Zhiqi, Jialin Mao, and Shiyun Xu. "Scalable and efficient training of large convolutional neural networks with differential privacy." Advances in Neural Information Processing Systems 35 (2022): 38305-38318. [2] - Kurakin, Alexey, et al. "Toward training at imagenet scale with differential privacy." arXiv preprint arXiv:2201.12328 (2022). [3] - Nasr, Milad, et al. "Effectively using public data in privacy preserving machine learning." International Conference on Machine Learning. PMLR, 2023. [4] - Ganesh, Arun, et al. "Why is public pretraining necessary for private model training?." International Conference on Machine Learning. PMLR, 2023. [5] - Feldman, Vitaly. "Does learning require memorization? a short tale about a long tail." Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing. 2020. [6] - Mehta, Harsh, et al. "Large scale transfer learning for differentially private image classification." arXiv preprint arXiv:2205.02973 (2022). [7] - Mireshghallah, Fatemehsadat, et al. "Differentially private model compression." Advances in Neural Information Processing Systems 35 (2022): 29468-29483.

Questions

Can the framework be used to explain the reason behind DP being 'more difficult' for untrained models through the prism of memorization? Is there a more theoretically sound way to define the threshold of public data that is required for appropriate performance?

Rating

4

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

As outlined above: while the theoretical justification of the results is indeed interesting, there are no novel conclusions our takeaways from this work that we have not previously had from similar works in the area of DP training.

Summary

This paper addresses the challenge of DP pretraining, which has been limited due to performance degradation. The authors first analyze per-iteration improvements of the optimization process and then propose a novel DP continual pre-training strategy that uses a small amount of public data to mitigate the negative effects of DP noise. They then empirically demonstrate the effectiveness of their approach on large-scale vision models.

Strengths

- Conclusions are well supported by figures and illustrations. - The framework of analyzing improvements appears novel to me. - The experimental evaluations are extensive, and the improvements are significant.

Weaknesses

- The authors claim that *a certain amount of public data can mitigate deceleration*. However, Section 4 lacks discussion about deceleration. The given claim is that per-iteration improvement is increased, which is obvious given public data. Also, it's unclear why *limited* public data is emphasized, as the analysis does not involve public sample sizes. This makes the article somewhat incoherent, as the interesting theoretical analysis in Section 3 does not provide strong suggestions for the methodology. - The experiments lack some ablation studies to support the key methodological proposal. - Citations should be updated to published versions rather than arXiv preprints where possible. - In line 14, there's an extra "and".

Questions

Overall, this paper is interesting. I am happy to raise my score if some of the questions are addressed. - Can the authors provide some comments on the first weakness? - Section 3.3 is somewhat confusing. First, Remark 3.1 investigates the differences between pre-training and fine-tuning by comparing $B_{non-DP} G^{\top} H G$ and $tr(H\Sigma)$, while in fact, public per-iteration improvement is monotonic with respect to $B$. Why use a comparison between $B_{DP}$ and $B_{non-DP}$ for deriving the explanation? Also, can the authors more rigorously explain what is meant by "data-efficient"? - Moreover, the difference between public improvement (7) and private improvement (6) lies in the decelerator $\sigma^2 tr(H) / (Bc^2)$, which seems smaller for pre-training where the loss landscape is flatter and thus the curvature $tr(H)$ is smaller. What is wrong with this intuition? - It is observed that prediction accuracy is linear in $\log B$, for instance in Figure 4 in [1], while the statement here is that $B$ should be moderately chosen. Though these are not the same quantity, could the authors comment on the impact of $B$ on DP-SGD? - Based on analysis in Section 3, is there any off-the-shelf rule for choosing $B$? - Can the authors provide some ablation studies (can be on toy datasets) on the impact of $s$? [1] Tom Sander, Pierre Stock, and Alexandre Sablayrolles. Tan without a burn: Scaling laws of dp-sgd. In International Conference on Machine Learning, pages 29937–29949. PMLR, 2023.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations should appear in the main text.

Summary

This paper proposed a Hessian-based analysis of the per-iteration loss for DP-SGD and non-private training, and provide a theoretical explanation for slower convergence of differentially private training. Authors identified the *decelerator* compontent in per-iteraton loss improvement, associated with the gradient clipping and noising, providing a theoretical backing for differences between DP impact on pre-training vs. fine-tuning. Using the same framework, authors also suggest the derivation for the optimal batch size for private training under the fixed computation budget (i.e. fixed number of samples processed), which strikes the balance between lower per-sample noise and training slowdown associated with growing batch size. Based on this analysis above, authors suggest the impact of DP on convergence can be mitigated by public pre-training data. While using public pre-training data to complement DP fine-tuning has been explored before, authors emphasize the difference between fine-tuning and continual pre-training, provide a theoretical analysis for the optimal mixing proportion for private/public data, and evaluate their models after private continual pre-training on a downstream fine-tuning tasks.

Strengths

I believe this is a strong and well-written paper with valuable contributions to the field. * The paper is very clearly written, with all major claims supported by arguments, derivations, and graphs. It's easy to follow and it gets the point across. * The derivation of the decelerator component of the per-iteration loss is a valuable contribution. It provides a good explanation for the DP-SGD training dynamics, and provides a theoretical backing for the previously observed empirical results. Specifically Fig. 6 provides very interesting insights intro DP-SGD training which I haven't seen analyzed before. * The downstream findings (e.g. optimal batch size or optimal ratio of private/non-private data) could be impactful for the practical applications of DP-SGD.

Weaknesses

The major weakness of the paper (at least its theoretical part) is the assumption that per-sample gradient clipping does not introduce bias into the gradient approximation (line 116). This is a very strong assumption, and something that, to the best of my knowledge, is not generally accepted in the field (e.g. see [this paper](https://proceedings.neurips.cc/paper_files/paper/2020/file/9ecff5455677b38d19f49ce658ef0608-Paper.pdf)). To justify the assumption authors point to Fig. 1, which to me doesn't look like it supports their claim - there's a significant difference between vanilla optimizer and optimizer + clipping in pre-training. Additionally, the analysis in this paper assumes the oracle knowledge of matrices G (gradient expectaition) and H (Hessian). For practical applications where this is infeasible, it would be useful to look at, e.g. optimal batch size not only from an optimal loss perspective, but also take into account how well the batch gradient approximates the actual gradient G. Oh the evaluation side, I see a slight disconnect with the theoretical results. For instance, after the results in Sec. 4.1 it would be natural to explore different mixing ratios of public/private data in training - however authors only focus on a setup with fixed pre-training and continual training datasets. The results presented in Tables 3,4 and 5 do make a good case that the proposed approach is valid, but lack proper baselines. Comparisons are made either with non-private models, models trained on different dataset, or model trained for different number of epochs - contradicting the scenario with fixed compute budget. Authors do not report compute resources used for the experiments, which is especially relevant for reproducibility of the paper's results, as it works with Hessian matrices which can be very computationally expensive to compute.

Questions

* I think the formatting for Fig.5 is wrong - in the text you refer to "upper left" plot, while in the submitted version it's rendered as a single line of 4 plots * I would be interested in reading how did you compute the data for Fig.6 - did you explicitly compute full matrices `G` and `H` or used some approximations? * I don't fully understand why in Fig.5 (pre-training) the blue dashed line is linear, suggesting that $G^THG$ is constant. * In Table 2, a) does number of images include continual pre-training? and b) what does "non-privacy" column refer to? * What is the criteria for distinguishing continual pre-training from fine-tuning? For example, in your experiments you perform continual pre-training with a different objective than earlier pre-training (supervised vs non-supervised). Does it not justify the "fine-tuning" term?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Authors do not discuss limitations explicitly, but are very upfront about assumptions they make (e.g. fixed compute budget, assuming no bias from clipping, etc). Authors also briefly mention limitations in the checklist (but not the main body of the paper).

Summary

This paper provides a theoretical framework to analyze impact of various aspects (parameters) of DP training on the performance of resulting models. The framework uses hessian of per-sample gradient to compute per-iteration loss improvement. Using the framework, the paper shows how DP impacts performance of model in pre-training more than during fine-tuning, and suggests using public data pre-training as a remedy. Finally, based on the observations, the paper proposes a DP continual learning and shows how it can help improve the performance of upstream and downstream tasks.

Strengths

- Approach of the proposed framework can be useful to analyze many DP settings. - Interesting conclusions especially Implication 2.4 about batch size - Proposed continual learning approach seems practically useful in privacy sensitive settings - Few shot accuracy of the proposed approach is impressive

Weaknesses

- Some assumptions, especially about clip norm, need more clarification - Some of the theoretical claims could be paired with empirical evidence - Proposed continual pre-training approach needs better explanation

Questions

- It looks like the conclusions made in the paper rely heavily on the assumption about clip norm multiplier being the same for all per-sample grads. The assumption is because in Figure 1 performances of models with and without clipping are similar. But, why would this be always true? If clipping is very aggressive, it will introduce bias and performance will reduce much more than what we see in Figure 1. Maybe it is good to clarify the scope of the assumption. - In the “Per-iteration improvement of DP-SGD” it says that parameter updates are generally small. When is this true in practice? - In Table 3, why do performances of many fine-tuning cases reduce from DINO to Ours(eps=2)? I think the performance should always improve? At the same time, few-shot performance of proposed method is significantly higher than DINO in all cases. Can you clarify this difference?

Rating

7

Confidence

3

Soundness

2

Presentation

2

Contribution

3

Limitations

- Assumptions made in the work to set up the theoretical framework need better clarification in terms of the scope of theory. - It would be good to formally write an algorithm of DP continual pre-training, as I found section 4.2 a bit confusing.

Response to the rebuttal

I would like to thank the authors for their responses! I still have some points I would like to note. From the discussion with reviewer JCrd I am not sure I agree with your response to them: > the extent of the bias is less significant and hence ignorable, compared to the effect of DP noising This may very well be the case for the specific settings of your experiments, but I am not convinced this is a general 'rule' you can follow and claim that this bias is 'ignorable'. > and noise reduction (e.g. via tighter privacy accounting in Implication 2.3) When I said 'actionable results', telling the community to use better DP tools (i.e. tighter accountant) is not really something actionable from YOUR results (i.e. we would always use the tightest accounting all else aside anyway). After reading the rest of the response, I am keeping my score unchanged.

Thanks for the response!

Thanks for the response. Rebuttal clarifies my questions and concerns, and i have raised my score accordingly.