Thank you for the follow-up questions. We answer them one-by-one below.
---
> Q1: Learning rate choice
We thank the reviewer for the interesting question.
(1) The main effect of the learning rate on the theoretical results in Jacot et al. [a] is the equivalence between the neural network and the linear model (that is, the speed at which the neural network converges to the linear model). For detailed results, refer to [b, Section F] and [c, Section 4] . This **does not materially affect the privacy utility results** for linearized networks in our paper, as we explain in (2) and (3) below.
(2) Our main privacy utility analyses focus on the Langevin diffusion algorithm, which is a conceptual algorithm that serves as the limit of noisy GD as learning rate goes to zero. Therefore, our comparison for privacy utility trade-off in Table 1 **does not involve setting any learning rate for the algorithm** (which may be why the reviewer says our techniques do not take into account the learning rate of the optimization).
(3) More importantly, our privacy and utility analysis extends seamlessly to a time-scaled Langevin diffusion algorithm, and **adding the time-scaling constant would not affect the privacy utility trade-off results** in Table 1. This is because, under changed time scaling, the lazy training distance constant $\tilde{R}$ does not change as the parameter space is the same, and the gradient norm constant $B$ and the loss scales up by a factor of $a$. This leads to the same excess empirical risk bound under fixed KL privacy budget as our current Corollary 5.4. Below we elaborate the details.
- Consider a time-scaled Langevin diffusion algorithm: $dW_t = - a \nabla L(W_t; D)dt + \sqrt{2\sigma^2}dB_t$, where $a$ is a time-scaling constant that plays a similar role as learning rate in discrete noisy GD.
- Observe that this time-scaled SDE is equivalent to Langevin diffusion Equation (4) if we replace $L(W; D)$ in Equation (4) with a new loss $\tilde{L}(W;D) = a \cdot L(W; D)$.
- Therefore, to prove excess risk bound for time-scaled Langevin diffusion, we only need to use our Corollary 5.4 for the new scaled loss $\tilde{L}(W;D) = a\cdot L(W; D)$. This gives us
$$
\mathbb{E} [\tilde{L}({W}_{T}^{lin})] \leq \mathcal{O}\left(\frac{{\color{red}a}}{n^2} + \sqrt{\frac{{\color{red}a}\cdot B \tilde{R}}{\varepsilon n }} \log (\varepsilon n)\right),
$$
where we have highlighted the change due to the loss rescaling by time-scale constant $a$. By rewriting $\tilde{L} = a\cdot L$, we immediately obtain exactly the same excess empirical risk bound as Corollary 5.4. That is, adding the time-scaling constant $a$ would not affect the privacy utility trade-off result in Table 1.
Accordingly, our comparison of privacy utility trade-off under different initialization is fair. We will add these clarifications in future revisions of the paper.
---
> Q2: NTK initialization formulation
Our initialization formulation follows Allen-Zhu et al. [2, Definition 2.3, arxiv v5], and the NTK initialization in [d] uses $N(0,1)$ and the scaling factor $1/\sqrt{m}$ at each layer (except the last layer). Allen-Zhu et al. [2, Theorem 5] proves the equivalence between these two different initializations.
---
We hope our response will clarify the reviewer’s concern on learning rate and initialization.
**References**
[2] Allen-Zhu, Z., Li, Y., & Song, Z. (2018). A Convergence Theory for Deep Learning via Over-Parameterization. arXiv preprint arXiv:1811.03962.
[a] Jacot, A., Gabriel, F., & Hongler, C. (2018). Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31.
[b] Arora, S., Du, S. S., Hu, W., Li, Z., Salakhutdinov, R. R., & Wang, R. (2019). On exact computation with an infinitely wide neural net. Advances in neural information processing systems, 32.
[c] Du, Simon S., et al. "Gradient descent provably optimizes over-parameterized neural networks." arXiv preprint arXiv:1810.02054 (2018).