Summary
This paper studies the training dynamics of PINNs, especially for the nonlinear PDEs. The authors find that the previous recognized NTK viewpoint is not applicable to nonlinear PDEs, although it holds for linear PDEs. Therefore, the global convergence of gradient descent on nonlinear PDEs may not be guaranteed. Moreover, the imbalance of singular values of Gram matrices, which also occurs in linear PDEs, results in slow convergence. To address this issue of spectral bias, the paper suggests using second-order methods for parameters update. Experimental results show that LM Newton’s method can achieve lower training loss compared to Adam and L-BFGS.
Strengths
The paper is well-written and provides both theoretical and empirical analyses. The failure of NTK approach on nonlinear PDEs has not been highlighted in previous studies of the global convergence of training PINNs with gradient descent. The second-order method is essential in smooth problems and deep learning training. The authors highlight the effectiveness of Newton’s method in balancing singular values of the training dynamics.
Weaknesses
(1) the failure of NTK is similarly investigated in some previous works (e.g., https://www.sciencedirect.com/science/article/pii/S016727892300341X), where they found that the Gram matrix does not consistently converge in some cases. Therefore, the observation seems to be not novel. They also pointed out that NTK may also work for some nonlinear PDEs.
(2) the second-order methods for modifying the singular values of Gram matrix of training dynamics are also not new.
(3) the Hessian inverse is quite expensive in practice.
(4) regularizing the Hessian may also be intractable in practice, for high dimensional problems, e.g., training PINNs.
Questions
I have the following questions:
(1) In the previous work (e.g., https://www.sciencedirect.com/science/article/pii/S016727892300341X), it was found that NTK approach fails in solving some PDEs. Besides linear PDEs, it is still possible that NTK works for some nonlinear PDEs. However, in your paper, you exclude the special situation. It seems to be more restrictive than the published work.
(2) Although the second-order methods enjoy very good theoretical properties, deep learning community typically prefers first-order methods due to their computational efficiency. In practice, Newton’s method is more computational expensive even with some inexact technique (e.g., Krylob subspace, conjugate gradient, and LBFGS). However, your theorem of global convergence (Theorem 4.2) does not apply to these inexact methods. Based on your result (which I believe built upon some previous works), can you extend your findings to more practical inexact Newton’s method (e.g., replacing Hessian inverse by Krylov subspace method or Quasi-Newton’s method of BFGS or LBFGS)?
(3) In your experimental results, the LM method performs significantly better than LBFGS. This is surprising, as LBFGS or BFGS asymptotically approaches the exact Hessian under certain conditions. Moreover, in case where the totally number of grid points are large and the batch is small in a stochastic setting, inexact and Quasi Newtons method (e.g., Hessian averaging methods including BFGS and LBFGS) should hold the asymptotic convergence. Therefore, intuitively, I would expect LBFGS to perform at least comparably with your LM method. Am I correct? Can you explain why LBFGS performs poorly according to your results, although it can approximate Hessian and extract Hessian information?