Initialization Matters: Privacy-Utility Analysis of Overparameterized Neural Networks

We analytically investigate how over-parameterization of models in randomized machine learning algorithms impacts the information leakage about their training data. Specifically, we prove a privacy bound for the KL divergence between model distributions on worst-case neighboring datasets, and explore its dependence on the initialization, width, and depth of fully connected neural networks. We find that this KL privacy bound is largely determined by the expected squared gradient norm relative to model parameters during training. Notably, for the special setting of linearized network, our analysis indicates that the squared gradient norm (and therefore the escalation of privacy loss) is tied directly to the per-layer variance of the initialization distribution. By using this analysis, we demonstrate that privacy bound improves with increasing depth under certain initializations (LeCun and Xavier), while degrades with increasing depth under other initializations (He and NTK). Our work reveals a complex interplay between privacy and depth that depends on the chosen initialization distribution. We further prove excess empirical risk bounds under a fixed KL privacy budget, and show that the interplay between privacy utility trade-off and depth is similarly affected by the initialization.

Paper

References (59)

Scroll for more · 38 remaining

Similar papers

Peer review

Reviewer R6b65/10 · confidence 4/52023-06-25

Summary

This work aims to connect the structure of a neural network and its initialization scheme to the privacy guarantees it can offer. KL divergence is used as privacy metric, under the settings of Langevine diffusion.

Strengths

The paper aims to solve a very important problem, i.e. the effects of overparameterization on privacy for overparametrized deep learning models. The analysis through KL divergence allows the problem to be treated analytically, and to connect the structure of the network to privacy bounds.

Weaknesses

fifth line of caption of Table 1: $n$ and $\phi$ appear here for the first time. It looks like they represent the number of training samples and your regularity condition parameter respectively. Probably better to mention this in the caption as well. same line: $m = \Omega(n^{14} L^{16} \phi^{-4})$. To me this assumption looks severely irrealistic for any deep learning model. The trade-offs in the table are simply upper-bounds. It is unclear to me how these upper-bounds can be considered tight, therefore telling anything about a possible comparison between different initializations / widths. Activation function and regularity parameter have the same notation $\phi$ Assumption 2.1 enforces the samples to respect $\| x_i\|_2 \leq 1$. This scaling is quite innatural considering your initializations schemes. which are designed to renormalize the preactivations when the input norms scale with $\sqrt d$ (more natural assumption). End of line 151, are the subscripts a typo here? It looks that the main bound in Lemma 4.1 derives from a worst case, per-layer computation, under the assumption of ReLU activation. This suggests the bound to be non tight, and therefore to not be informative about how different initializations really affect the preservation of privacy. Lack of experimental validation on the claims in the paper.

Questions

third line of caption of figure 1: you write $d = m$ first, and then you mention that you assume $m \geq d$. Is this a typo? would your results still hold if we consider a more natural scaling of the data $\| x_i\|_2 \leq \sim d$? Your results are valid only in the case of ReLU networks. Is the analysis generalizable? In the discussion after Lemma 3.2 you mention that there are two important keys to control privacy: initialization and training time. You focus your analysis on the first one, and you motivate why the second component is not relevant in the case of small training time. Correct? If yes, how do you verify this hypothesis? Is it a consequence of lazy training? Your Lemma 4.1 looks to exploit a worst-case bound over the layers of the network, plus the property of the ReLU activation. Can you confirm?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

No limitations discussion needed

Reviewer NB2b7/10 · confidence 4/52023-07-06

Summary

This paper presents the KL privacy bound for training fully connected networks and its linearized variant using the Langevin diffusion algorithm. The study delves into the impact of network width, depth, and initialization on privacy. Notably, the paper reveals that, with specific parameter initialization, the KL privacy bound can be enhanced by increasing the depth of the network. Additionally, the author provides a proof that the privacy-utility trade-off of linearized networks can be improved with depth when employing LeCun initialization.

Strengths

1.The author investigates an intriguing problem that is likely to capture significant interest within the community. 2.The results presented in the paper are not only interesting but also coherent, aligning well with the research topic. 3.The paper is skillfully written, allowing for easy comprehension and smooth flow of ideas.

Weaknesses

The paper lacks numerical experiments to validate the disparity between empirical KL privacy leakage and the privacy bound proposed in this paper. This would shed light on the tightness of the bound. Given that the arguments are based on the upper bound, providing numerical experiments demonstrating the evolution of empirical KL privacy leakage with varying depths and widths, aligned with the theoretical findings, would enhance the persuasiveness of the research.

Questions

Questions: 1. In Lemma 3.2, Equation (7) suggests that for relatively large T, the dominant factor in the KL privacy bound is the fluctuation term of gradient differences, which appears to be independent of the differences between neighboring datasets $D$ and $D'$. Could the author provide further clarification on this matter? 2. Additionally, I am curious about the evolution of the rank of $M_T$ with respect to time T. The author briefly touches upon this aspect, but a more detailed explanation would be appreciated. Some suggestions and corrections for the paper: 1. In Assumption 2.2, it would be preferable to use a commonly recognized constant symbol such as 'C' instead of '\phi' to denote a constant. 2. There seems to be some confusion in the notations in lines 151 and 152. Is it intended that x_1 = x, y = y_1, and y' = y_2?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See above.

Reviewer Ej4s5/10 · confidence 3/52023-07-08

Summary

This paper theoretically analyzes how the width, depth, and initialization of deep neural networks affect the privacy utility trade-off. For the special setting of linearized networks, the paper shows that gradient norm and privacy loss hinge on the per-layer variance of the initialization distributions. Thus, the privacy trade-off can be governed by width, depth, and initialization. Interestingly, for LeCun's initialization, the utility can be amplified by increasing depth which is against common intuition that overparameterized models will worsen the privacy utility trade-off.

Strengths

1. The theoretical finding on the connection between depth and privacy-utility trade-off is interesting, that the deeper network can achieve better tradeoff than the shallow ones given LeCun's initialization. 2. It is novel that the network properties, e.g., initialization is analyzed through the Lagevian dynamics on linearized networks.

Weaknesses

1. The presentation coould be improved. Some typos in the formulations make the paper hard to follow. * KL privacy bound is used before the definition or explanation in Line 38. Especially, what is the relation between the KL privacy bound and differential privacy bound? * The definition of the non-lazy training regime in Table 1 is not defined. * For the definition of LeCun initialization, the $\beta$ is layer-independent in Line 262 as $\beta_2=\cdots =\beta_L={1\over m}$. But in Table 1, the $\beta$ is layer-dependent as $\beta_l = \frac{1}{m_{l-1}}$. * In Line 24, `is randomze` should be `is randomizing`. 2. The difference between LeCun's initialization $\beta_l=1/m_{l-1}$ and others $\beta_l=2/m_{l-1}$ is mainly in the factor, `2`. The authors could give a more intuitive explanation for why the factor is crucial for privacy. 3. More generally regarding the factor in initialization, analyzing the best factor (probably some value between 0 and 2) may lead to a counterintuitive trade-off between privacy and utility. If I follow the derivations in Line 264-265, the excess empirical risk for LeCun's initialization with $\beta_t=\frac{a}{m_{l-1}}$ with $a\in (0,2)$ could be proportional to $\frac{a}{2}^{L/2}$. This means that the risk bound will be greatly reduced when $a$ approaches zero. However, this is counterintuitive, as a smaller $a$ implies smaller $\beta_t$, and therefore model will be initialized almost as zero. As common knowledge, zero parameters are not favored for the initialization of a deep network. In brief, smaller initial parameters will quickly vanish the activations by layer and therefore the model cannot converge or of low utility. Unless I missed some assumptions, the derivation is problematic due to the above counterintuitive results.

Questions

1. What is the relation between the KL privacy bound and the differential privacy bound? What is the motivation to study the KL privacy bound? 2. Can you explain the counterintuitive derivation given a variable factor in $\beta_l$? See Weakness 3 for details. ----- Update after reading rebuttals --- The authors have addressed most of my concerns.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors did not discuss the limitations or potential social impacts of the paper. 1. The paper presents some promising privacy improvement through proper initialization and also demonstrate some increasing risks if networks are not properly initialized. The results could be an important signal for people to treat the initialization seriously. I suggest the authors make the social impact clearly in the paper. 2. The paper has some inherent limitations due to the analysis tools of Lagevian diffusion algorithms, for example, the requirement for linearized networks, large depths, etc. The limitations are hard to address in the scope of the paper but is worth to mention for readers to under the proper use cases.

Reviewer LMCn6/10 · confidence 2/52023-07-09

Summary

The authors propose a KL privacy bound for DP-SGD that depend on the width, depth, and initialisation of the neural network. They show that the privacy bound improves with increasing depth for the LeCun and Xavier initialisation while it degrades with increasing depth.

Strengths

1. The authors came up with an original idea exploring a topic that has so far received little attention in the literature 2. the findings are somewhat surprising and unintuitive

Weaknesses

1. Often there are mistakes in privacy proofs. Having some experiments with privacy auditing to back up the surprising results would make the results more credible. 2. There are many typos in the middle of the paper. Especially related to the use of articles (articles missing) or repetitive words. 3. The authors motivate their contribution by the loose privacy bounds of DP-SGD. But the privacy bounds are then not used to improve the privacy accounting of the neural network. 4. Limitations not discussed in depth

Questions

1. The result about the improving privacy bound is really surprising. Do the authors have an intuition why this is the case?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

Limitations are not explicitly mentioned. The authors touch upon the restrictive results on linearised networks but this limitation is mostly underplayed. It would be interesting to discuss more when this simplification can lead to worse results.

Reviewer R6b62023-08-13

I thank the authors for their detailed responses. I appreciate the introduction of experiments in support of your bounds, reason for which I increased my previous score. I have three follow-up questions: 1 - Regarding Equation (10), this is an upper-bound. In principle, taken as such, it wouldn't give any information about which initialization would provide smaller KL divergence. However, if considered tight, it might suggest the conclusions you draw at the end of Section 4, after Corollary 4.2. Now, with the new numerical evaluation (in the new experiments), this hypothesis is more supported. Is this correct? If so, I would really suggest to modify the presentation in the revised version, as the word "suggest" at line 257 is at the moment not very motivated. 2 - How do you compute the KL divergence in your experiments? 3 - KL divergence between two distributions in high dimension is expected to increase with the number of dimensions (not a very precise statement, but we would expect divergence to increase as more degrees of freedom are offered to the evolution of the weight distribution). This makes it reasonable to have smaller privacy guarantees (with this metric) for more over-parametrized networks. Can you elaborate on your contribution stated in line 44 based on this?

Authorsrebuttal2023-08-15

Replies to the follow-up questions

Thanks for the reply and detailed questions. We answer them below. > Q1: Eq (10) and numerical results Yes, equation (10) is an **upper bound** for gradient difference **throughout training**. Meanwhile, our gradient norm computation **at initialization** uses an equality (equation (35) in Lemma C.3), and is thus **exact**. Our numerical experiments validate the exact gradient norm computation at initialization, and are consistent with the trend predicted by Lemma 4.1 at small training time (thus supporting its tightness). According to your suggestions, we will make our presentation precise in the revised version, e.g., “Corollary 4.2 and Lemma 4.1 prove that the KL privacy upper bound ...". Besides, we will clarify that this trend of upper bound is consistent with numerical experiments at small training time. > Q2: How do you compute the KL divergence in your experiments? Our detailed numerical estimation procedures for noisy GD with $K$ steps, noise std $\sigma$ and learning rates {$\eta_k$}$_{k=1,\cdots,K}$ are as follows. 1. At each noisy GD iteration $k$, we **compute the maximal squared gradient difference norm** $s(k)$ between all possible neighboring datasets in CIFAR10. (See [general response point 1](https://openreview.net/forum?id=IKvxmnHjkL&noteId=jh61g9pgBi) for details about neighboring datasets.) We repeat this for $N$ noisy GD runs, and obtain $N$ squared gradient difference norm estimates $s_1(k), \cdots, s_N(k)$ (with mean $\hat{s}(k)$ and standard deviation $std(k)$). 2. We then plug the squared gradient norm into the closed-form KL divergence equality $\frac{\lVert \text{offset} \rVert_2^2}{2\sigma^2}$ (e.g. see [a, Proposition 7]) between isotropic Gaussian and its offset (by learning rate $\eta_k$ times gradient difference $\hat{s}_k$) to **compute the one-step KL privacy loss for each iteration $k$**. This gives us mean $\frac{\eta_k\cdot \hat{s}(k)}{2\sigma^2}$ and std $\frac{\eta_k\cdot std(k)}{2\sigma^2}$. 3. Finally, we **compose (sum) KL divergence over all iterates** $k=1, \cdots, K$ of noisy GD, via the composition **equality** Equation (89). This gives us mean $\frac{\sum_{k=1}^K\eta_k\cdot \hat{s}(k)}{2\sigma^2}$ and std $\frac{\sum_{k=1}^K\eta_k\cdot std(k)}{2\sigma^2}$ for estimating the KL privacy loss in noisy GD, which we plot in numerical experiments. > Q3: KL divergence may not grow with the distribution dimension We guess the reviewer had a typo and meant “unreasonable”. To check this claim, it is valuable to first look at one simple example of KL divergence between two $d$-dimensional Gaussians $\mathcal{N}(\mu_1, \sigma^2I_d)$ and $\mathcal{N}(\mu_2, \sigma^2I_d)$, which is $\frac{\lVert \mu_1 - \mu_2\rVert_2^2}{2\sigma^2}$. Therefore, dimension does not factor in KL as long as $\lVert \mu_1 - \mu_2\rVert_2^2$ is small. We can show that the gradient difference (in noisy GD for overparameterized linearized network) achieves this under certain initializations (such as LeCun and Xavier), therefore our KL divergence bound (Corollary 4.2) **does not necessarily grow with the model dimension (increasing width and depth)**, as summarized in our contributions Line 44-52. We elaborate them below. (1) On the one hand, **increasing width $m$ always increases the gradient difference norm** ($B$ in Table 1) between the two distributions, and thus increases the KL divergence bound (Corollary 4.2). Intuitively, this is because the gradient at each layer is proportional to **the weighted sum of outputs from its preceding layer**, and thus increases with larger hidden-layer width. This is **consistent with intuition that the reviewer quoted**. (2) On the other hand, if we inverse proportionally set the per-layer initialization variance $\beta_l$ with the width $m_l$ (such that $\frac{\beta_lm_l}{2}<1$), and increase the network depth, then the gradient difference norm ($B$ in Table 1) and thus the KL privacy bound (Corollary 4.2) decreases. Intuitively, this is because of the output propagation across the layers in the network. For example, under LeCun initialization, the output values after each layer gets zeroed out with 1/2 probability. Consequently, the gradient norm at each layer decays exponentially with the number of preceding layers (depth). Such decrease of gradient norm under increasing depth is not a new phenomenon, and has been widely referred to as “gradient vanishing” in the literature. To this end, our contribution includes a general bound for the gradient norm (Lemma 4.1), under different initialization, and analyzing when it does not necessarily grow with model dimension. We'd also like to refer to [general response point 1](https://openreview.net/forum?id=IKvxmnHjkL&noteId=jh61g9pgBi) for more discussion about the intuition for this gradient norm bound. We hope our response can address the reviewer’s concern and are also happy to address any further comments. **Reference** [a] Mironov, Ilya. Rényi differential privacy. CSF 2017.

Reviewer R6b62023-08-17

I warmly thank the authors for their detailed response, which also clarified better parts of the work. It looks from your techniques that you are not taking into account the learning rate of the optimization. Different initializations (e.g. NTK vs LeCun) would reasonably require different scalings of the learning rate (see Remark1 of https://arxiv.org/pdf/1806.07572.pdf). Thus, I wonder if your results might have roots in a not fair comparison between two different initializations, as they would need to be compared with the properly rescaled learning rates. Also, I am a little confused on the NTK initialization you consider in Table1, as it seems different from the one defined in https://arxiv.org/pdf/1806.07572.pdf. Could you clarify this?

Authorsrebuttal2023-08-18

Thank you for the follow-up questions. We answer them one-by-one below. --- > Q1: Learning rate choice We thank the reviewer for the interesting question. (1) The main effect of the learning rate on the theoretical results in Jacot et al. [a] is the equivalence between the neural network and the linear model (that is, the speed at which the neural network converges to the linear model). For detailed results, refer to [b, Section F] and [c, Section 4] . This **does not materially affect the privacy utility results** for linearized networks in our paper, as we explain in (2) and (3) below. (2) Our main privacy utility analyses focus on the Langevin diffusion algorithm, which is a conceptual algorithm that serves as the limit of noisy GD as learning rate goes to zero. Therefore, our comparison for privacy utility trade-off in Table 1 **does not involve setting any learning rate for the algorithm** (which may be why the reviewer says our techniques do not take into account the learning rate of the optimization). (3) More importantly, our privacy and utility analysis extends seamlessly to a time-scaled Langevin diffusion algorithm, and **adding the time-scaling constant would not affect the privacy utility trade-off results** in Table 1. This is because, under changed time scaling, the lazy training distance constant $\tilde{R}$ does not change as the parameter space is the same, and the gradient norm constant $B$ and the loss scales up by a factor of $a$. This leads to the same excess empirical risk bound under fixed KL privacy budget as our current Corollary 5.4. Below we elaborate the details. - Consider a time-scaled Langevin diffusion algorithm: $dW_t = - a \nabla L(W_t; D)dt + \sqrt{2\sigma^2}dB_t$, where $a$ is a time-scaling constant that plays a similar role as learning rate in discrete noisy GD. - Observe that this time-scaled SDE is equivalent to Langevin diffusion Equation (4) if we replace $L(W; D)$ in Equation (4) with a new loss $\tilde{L}(W;D) = a \cdot L(W; D)$. - Therefore, to prove excess risk bound for time-scaled Langevin diffusion, we only need to use our Corollary 5.4 for the new scaled loss $\tilde{L}(W;D) = a\cdot L(W; D)$. This gives us $$ \mathbb{E} [\tilde{L}({W}_{T}^{lin})] \leq \mathcal{O}\left(\frac{{\color{red}a}}{n^2} + \sqrt{\frac{{\color{red}a}\cdot B \tilde{R}}{\varepsilon n }} \log (\varepsilon n)\right), $$ where we have highlighted the change due to the loss rescaling by time-scale constant $a$. By rewriting $\tilde{L} = a\cdot L$, we immediately obtain exactly the same excess empirical risk bound as Corollary 5.4. That is, adding the time-scaling constant $a$ would not affect the privacy utility trade-off result in Table 1. Accordingly, our comparison of privacy utility trade-off under different initialization is fair. We will add these clarifications in future revisions of the paper. --- > Q2: NTK initialization formulation Our initialization formulation follows Allen-Zhu et al. [2, Definition 2.3, arxiv v5], and the NTK initialization in [d] uses $N(0,1)$ and the scaling factor $1/\sqrt{m}$ at each layer (except the last layer). Allen-Zhu et al. [2, Theorem 5] proves the equivalence between these two different initializations. --- We hope our response will clarify the reviewer’s concern on learning rate and initialization. **References** [2] Allen-Zhu, Z., Li, Y., & Song, Z. (2018). A Convergence Theory for Deep Learning via Over-Parameterization. arXiv preprint arXiv:1811.03962. [a] Jacot, A., Gabriel, F., & Hongler, C. (2018). Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31. [b] Arora, S., Du, S. S., Hu, W., Li, Z., Salakhutdinov, R. R., & Wang, R. (2019). On exact computation with an infinitely wide neural net. Advances in neural information processing systems, 32. [c] Du, Simon S., et al. "Gradient descent provably optimizes over-parameterized neural networks." arXiv preprint arXiv:1810.02054 (2018).

Reviewer R6b62023-08-20

I thank the authors for their explanation. I am satisfied, and I apologize again for the pedantic evaluation of the work in this review phase. I think further explanation about these points (maybe in the appendix) could be substantially improve the quality of the final version. I further increased my score, which I consider final.

Authorsrebuttal2023-08-21

We thank the reviewer R6b6's feedback and support. According to your suggestions, we will include these explanations in our main text and appendix for better understanding in our final version.

Reviewer Ej4s2023-08-14

Thank you for the responses

The authors' responses are helpful for my understanding of the paper and address my major concerns. I suggest the authors integrate them into the final version.

Authorsrebuttal2023-08-16

Thank you for the reply

We thank the reviewer for the positive reply. According to your suggestions, we will improve the presentation of this work and include the discussion in the revised version. We are happy that our clarifications addressed most of your concerns and please feel free to let us know if you have any remaining concerns or further questions.

Reviewer LMCn2023-08-18

Thank you for the detailed response. After reading the rebuttal and the other reviewers’ comments, I have raised my score.

Authorsrebuttal2023-08-21

Thanks for your support. In our final version, we will include the experimental validation and polish the language based on your suggestions.

Reviewer NB2b2023-08-21

Thanks for the detailed explanation.

The authors have provided a clear response that addresses my concerns. Consequently, I have adjusted my score upwards.

Authorsrebuttal2023-08-21

We appreciate your positive support on our work. According to your suggestions, we will clarify our theoretical results as well as numerical validations in the final version.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC