Global Convergence in Neural ODEs: Impact of Activation Functions

Thank you for your detailed review and for acknowledging the novelty of our analysis and results, particularly in extending NTK theory to Neural ODEs and ResNet architectures. We are also grateful for your recognition of the soundness of our results, the excellent presentation, and the significant contributions of our work. Below, we address your specific comments and weaknesses. > Weakness (1): Missing Related References We sincerely thank the reviewer for pointing out these important prior results on NTK theory. In the revised manuscript, we have incorporated references to NTK analyses for various architectures and methods, including CNNs, RNNs, PINNs, attention mechanisms, GNNs, and optimization techniques such as SGD and adaptive gradient descent. To further contextualize our contributions within the broader NTK literature, we have also included a detailed **Related Works section**. However, due to page limit, we moved this comprehensive discussion to the appendix. We believe this approach balances the need to present our main theoretical contributions in the primary text while properly acknowledging foundational work in the field. We greatly appreciate this suggestion, which has helped us enhance the clarity and completeness of our manuscript. > Weakness (2): Convergence Analysis for SGD We agree with the reviewer on the importance of analyzing SGD, as it is widely used in practice. While our study focuses on gradient descent (GD) as a foundational step, this is significant since no theoretical convergence analysis for Neural ODEs under any gradient-based method has been established. Given that SGD is an approximation of GD using mini-batches, our results provide a crucial basis for studying SGD and other advanced optimizers. We recognize the practical value of such analyses and consider extending our work to SGD as a natural and important direction for future research. > Weakness (3): Generalization Beyond Specific ODE Forms Thank you for raising this important point regarding the applicability of our theoretical results, such as Theorem 2, to more general ODEs or PINNs with nonlinear PDEs. While our current analysis assumes the specific ODE form in Eq. (2), we believe the results have the potential for extension to broader setups. To ensure such generalization, it is crucial to establish that the forward and backward dynamics remain **well-defined**, i.e., they have globally unique solutions and exhibit **uniform convergence** of the Euler method. To address this, we have added a new **Discussion Section** in the appendix of the revised version, where we explore the extension of our results to generalized ODE forms. In particular, we examine the pre-activation form: $$ \begin{align*} \dot{h} = \frac{\sigma_w}{\sqrt{n}} W f(h, t),\quad \forall t \in [0, T], \end{align*} $$ where $f:\mathbb{R}^{n}\times \mathbb{R}\rightarrow \mathbb{R}^n$ is a general nonlinear mapping that replace the activation function $\phi$ in Eqution (2). Here, the explicit time-dependence transforms the system from *autonomous* to *non-autonomous*, a framework widely used in PINNs and diffusion models. This general nonlinear mapping *f* encompasses components like shallow neural networks, layer normalization, attention, or gating mechanisms. For the generalized form, the backward dynamics become: $$ \dot{\lambda} = -\frac{\sigma_w}{\sqrt{n}} J(h, t) W^\top \lambda, $$ where $J=\partial f/\partial h\in \mathbb{R}^{n\times n}$ is the Jacobian of $ f $ w.r.t. $ h $. By Picard-Lindelöf Theorem and Euler's convergence theorem, the regularity conditions we require for activation functions $\phi$ in this paper must extend to the nonlinear map $f$. For example, ensuring the backward dynamics have a globally unique solution necessitates that $J$ is globally bounded. Furthermore, gradient alignment between the “optimize-then-discretize” and “discretize-then-optimize” methods additionally requires $J$ to be Lipschitz continuous in $h$, under suitable norms such as Frobenius or operator norms. In the Discussion Section, we also examine the post-activation form: $$ \begin{align*} \dot{h} = f\left(\frac{\sigma_w}{\sqrt{n}} W h, t\right), \end{align*} $$ where the linear transformation $ h \mapsto \frac{\sigma_w}{\sqrt{n}} W h $ is applied before the nonlinear mapping $ f $. The analysis remains analogous because the linear transformation is globally $ 1 $-Lipschitz continuous under Bai-Yin law from random matrix theory. However, we focus on the pre-activation form due to its superior empirical performance [1] compared to the post-activation formulation. We hope this discussion addresses the broader applicability of our theoretical framework and highlights potential extensions to more general ODEs and PINN setups. Thank you for encouraging us to elaborate on this important aspect of our work. [1] Identity mappings in deep residual networks, ECCV2016

Thanks for the rebuttal. I am fully satisfied with your responses and the updated version of the paper (including the title modification). I am pleased to raise my scores to 8.

Thank you very much for your thoughtful review and high evaluation of our work. We deeply appreciate your recognition of the novelty and significance of our contributions, as well as your positive assessment of the theoretical foundation and empirical validation. Your feedback is incredibly encouraging and motivates us to continue refining our work.

We sincerely thank the reviewer for the detailed feedback and for acknowledging the theoretical rigor and contributions of our work, including the NTK analysis, global convergence proof, and empirical validation. Below, we address each of the points raised. > Lack of Demonstration for “Discretize-then-optimize” Convergence Issue We appreciate your observation regarding the need for a dedicated numerical experiment to demonstrate the convergence issue of the “Discretize-then-optimize” method with *non-smooth* activation functions. To address this, we have conducted additional experiments, as detailed in *Appendix H.6* of the revised manuscript. Specifically, we compare the output and gradient differences between the “Discretize-then-optimize” and “Optimize-then-discretize” frameworks using ReLU (non-smooth) and Softplus (smooth) activations. Our findings, illustrated in Figure 12, show that while the output differences reduce with increasing depth L, the gradient differences for ReLU fail to converge, confirming our theoretical claims. These results underscore the importance of continuous differentiability for gradient alignment, as discussed in the main text. Interestingly, despite the gradient mismatch, we observed no significant differences in training dynamics between Neural ODEs and their finite-depth ResNet approximations. In experiments where both models were trained (ResNets fixed at depth 200, as further depth increases did not reduce errors, as shown in Figure 12(d)) on a subset of MNIST, their training and test losses were nearly identical, as illustrated in Figure 13. While output differences remained consistently small, gradient differences oscillated during training. ResNets, as finite-depth networks, are known to achieve global convergence under gradient-based methods in overparameterized regimes [1], making their behavior expected. However, the near-identical training dynamics of Neural ODEs and ResNets, despite the gradient mismatch caused by ReLU’s non-smoothness, were surprising. We hypothesize that the small magnitude of gradient differences, attributed to MNIST’s simplicity and ReLU’s near-continuous derivative (except at the origin), mitigates the adverse effects of the mismatch. Nevertheless, we anticipate that with more complex datasets, these gradient discrepancies could lead to divergent training trajectories. > Sensitivity to Numerical ODE Solvers Your suggestion to explore the sensitivity of the “Optimize-then-discretize” method to different numerical ODE solvers is highly valuable. In response, we have included a new experiment comparing solvers such as Euler, RK4, and Dopri5 in Appendix H.7. Our results show that while all solvers maintain consistent accuracy due to the well-posedness of the ODE dynamics (as proven in Proposition 1), the computational time varies significantly. Adaptive solvers like Dopri5 are slower and scale poorly with increasing neural network width, whereas fixed-step solvers like Euler and RK4 are more scalable. These findings highlight the trade-off between solver choice and computational efficiency in practice. > Practical Implications for Activation Function Selection We agree that providing practical guidelines for activation function selection is important. Our results suggest the following: - Smooth and Non-Polynomial Activations: Smooth activations (e.g., Softplus) ensure gradient alignment and faster convergence, making them preferable for applications requiring high accuracy and robust optimization. - Non-Smooth Activations: Activations like ReLU may lead to convergence issues in gradient alignment, as demonstrated in Figure 12. Thus, they are less suitable for scenarios requiring precise gradient computations. - Polynomial Activations: While our experiments in Figure 8 (Appendix H.4) show that polynomial activations (e.g., quadratic) maintain the SPD property of NTK, their slower convergence suggests they may be less effective in practice. [1] Gradient descent finds global minima of deep neural networks. ICML 2019

We appreciate your thoughtful comments and your acknowledgment of the paper’s strengths, including precise and rigorous results, clear presentation, and the extension of NTK theory to Neural ODEs using classical analytical tools. Your feedback provides valuable insights to refine our work further. Below, we address your points systematically. > Typographical Error in Equation 1. Thank you for pointing out the typo in the index of $h_t$ in Equation 1. You are correct that the last $h$ should have the subscript $T$ instead of $t$. We have corrected this in the revised manuscript. > Log-Log Scale for Figure 1 We really appreciate your suggestion to plot Figure 1 using a log-log scale to better highlight the 1/L convergence rates. This is an excellent idea, and we have updated the figure accordingly to improve clarity and better emphasize the convergence rates. Please see the update in the revised version. > Title Suggestion and Focus on Activation Impact We understand your concern about the alignment between the title and the paper’s primary focus. The current title may overemphasize the impact of activation functions, potentially overshadowing the core theoretical contributions. To address this, we are open to revising the title in the final version to better reflect the paper’s focus and avoid any confusion. That said, the impact of activation functions is an integral part of our study. Due to space constraints, we included comprehensive experimental analyses on activation functions in the appendix. For instance, as shown in Figure 12 (Appendix), when the activation function is nonsmooth (e.g., ReLU), the gradient differences between the “discretize-then-optimize” and “optimize-then-discretize” frameworks fail to converge as depth increases. This suggests that continuous differentiability is not just sufficient but likely necessary for gradient alignment. Additionally, we investigated the SPD property of NNGP and NTK under polynomial activation functions, such as quadratic activation, in Figure 8 (Appendix). Our experiments show that quadratic activations still ensure SPD. Hence, the non-polynomial nonlinearity seems to be a sufficient but not necessary condition. Moreover, we also observed that the convergence is significantly slower compared to non-polynomial nonlinear activations like Softplus. This highlights the practical advantage of non-polynomial activations in accelerating training. In summary, while the theoretical contributions form the core of the main paper, the experimental results offer valuable insights into the implications of activation regularity. These findings complement the theoretical results and address broader questions about activation design. Thank you for highlighting this balance, and we will ensure the final version reflects this alignment more clearly.