Dual Cone Gradient Descent for Training Physics-Informed Neural Networks

Summary

**Summary:** The paper proposes viewing the optimization of PINNs as a multi-objective optimization problem, with boundary and residual terms as potentially conflicting objective functions. To address conflicting gradients in the optimization process, the dual cone of the cone generated by the gradients of the boundary and residual terms is designated as the set of non-conflicting update directions. Several algorithms that guarantee updates from this dual cone are proposed, and their empirical performance is assessed. **General Impression:** The manuscript is well-written, easy to follow, and generally of high quality. The numerical experiments are convincing and demonstrate the good performance of the proposed method. However, I disagree to a certain extent with the multi-objective viewpoint for the optimization of PINNs. For well-posed forward problems, the boundary loss and the interior loss are not really conflicting losses—at least if we assume that the network ansatz is sufficiently expressive. I expand more on this issue below in the questions. Nevertheless, the preprint presents a convincing suite of numerical experiments that seem to benefit from the multi-objective optimization approach.

Strengths

**Strengths:** 1. Considering the dual cone of the interior and residual gradients is a simple and sound idea. The presentation is clear and understandable. 2. The section on numerical results demonstrates the benefits of the proposed method and incorporates many recently proposed techniques for the optimization of PINNs.

Weaknesses

See questions.

Questions

**Questions** 1. It is unclear whether PINN optimization should be viewed as a multi-objective optimization problem. Please comment. 2. I am missing a clarification on whether the proposed dual cone gradient descent is a completely novel idea, even in the field of multi-objective optimization, or if it is well-known in that field and its application to PINNs is novel. Please clarify and give references if appropriate. 3. Could the authors comment on how their proposed method relates to loss reweighting strategies? It seems that the proposed approach could equivalently be understood as a loss reweighting strategy. 4. Can the authors comment on the case of more than two loss functions. 5. Have the authors considered measuring angles in function space instead of parameter space by using the differential of the parametrization map? Recently, optimization methods that employ function space geometry have shown promise for the optimization of PINNs (see [1, 2, 3]). This might lead to further improvement of the proposed method. **Details to question 1:** A fundamental question for me is whether to view PINNs as a multi-objective optimization problem or not. To illustrate this question, consider the following academic example of a PDE with homogeneous forcing and inhomogeneous boundary data: \begin{align} Nu &= 0 \quad \text{in }\Omega, \\ \newline Bu &= g \quad \text{on }\partial\Omega. \end{align} Assume further that $N$ maps constant functions to zero (any PDE operator without a zeroth-order term). Assume a trial neural network is initialized representing (almost) the zero function (or any other constant function). Any change in the neural network will now likely increase the PDE residual (which is at its global minimum for any constant function). Furthermore, the increase in the PDE residual is necessary to obtain the correct solution. This might lead to a Pareto stationary point that has nothing to do with the sought-after solution. **References** [1] https://proceedings.mlr.press/v202/muller23b/muller23b.pdf [2] https://openreview.net/pdf?id=z9SIj-IM7tn [3] https://arxiv.org/pdf/2402.07318

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

See questions.

Summary

The paper concerns with the training of physics-informed neural networks (PINNs). It observes that in a multi-objective setting a naive gradient update which decreases the total objective might not decrease every individual objective. This is used to explain the challenge of training PINNs which have a potentially conflicting PDE and a boundary-condition loss. As a remedy, the work proposes to find another descent direction which decreases both of the PINN losses. Three different methods are proposed and the superior "center" method is selected for more detailed empirical investigation. It demonstrates strong results relative to several existing PINN optimizers.

Strengths

The paper presents a clear central story that explains multi-objective optimization in terms of simple geometric concepts. The theoretical analysis in addition to the empirical evaluation is a welcome addition.

Weaknesses

However, the significance of the theoretical contributions must be questioned. There are just two theorems, of which 4.2 is fairly trivial and 4.5 is unclear: why is $\frac{1}{T+1}$ on both sides of Equation 4.2, and what does this equation even state? I cannot make sense of the provided intuition that the convergence rate is $\mathcal{O}(1/T)$, i.e., the more iterations, the slower you converge. Even more so, it is unclear how this intuition follows from Eq. 4.2 in the first place: it says that the sum of gradient norms is upper-bounded by the gap from the initial to the optimal loss, up to some constant. The work also overlooks some recent literature explaining the difficult behavior of PINNs in terms of poor conditioning and proposing strong optimization strategies, which to the best of my knowledge achieve state-of-the-art convergence. I would argue these ideas should be included in the related work and as baselines contributing to a much more complete story. - *De Ryck et al. An operator preconditioning perspective on training in physics-informed machine learning. ICLR 2024.* - *Rathore et al. Challenges in Training PINNs: A Loss Landscape Perspective. ICML 2024.* Several presentation choices could be improved, including minor mathematical and typing errors, odd wording, and superfluous parts (see suggestions below). The work also overstates the practical success of PINNs: "a prominent approach for solving PDEs" (line 2), "remarkable empirical performance" (line 4), "PINNs have achieved great success in a wide range of applications" (line 8). This sets a misleading context for the practical significance of PINNs and by extension the presented work. - *Grossmann et al. Can Physics-Informed Neural Networks beat the Finite Element Method? IMA Journal of Applied Mathematics 2024.*

Questions

- I would kindly ask the authors the clarify Theorem 4.5 in terms of the aforementioned concerns. - How do the aforementioned optimization strategies compare to your experiments? Do you foresee a way for your method to be compatible to these (similar to section 5.3)? - Does your method extend to more than two losses? Even in PINNs, there might be additional losses for data and initial conditions. - How important is the assumption of fixing $\omega_r = \omega_b = 1$? Do the other optimizers show improved performance for other choices of these loss weights?

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

The limitations are discussed rather sparingly and should also include that the method currently applies to just two losses. It should also be noted that the performed experiments are closer to toy than practical problems. **Errors and suggestions** - Line 59: It would be helpful if you briefly described what each of these three approaches do, instead of only listing examples. - Line 73: $d$ is used for both the dimension of the domain as well as the parameter space. - Eq. 2.2 and line 88: $\omega_u$ should be $\omega_b$ - Line 79: should this be an equality? - I do not think writing out algorithm 1 adds much value as this is precisely gradient descent but with a different gradient. - Line 170: should there be an argmin or arginf instead of inf? Or alternatively $\mathcal{L}(\theta*)$ instead of $\theta*$ - Lines 229-240 and Figure 6 might better belong to the appendix as it detracts attention from the main story. - Line 185: "presents" -> "present" - Line 221: "ADAM" -> "Adam" - A.4: the first inequality should be an equality - Adjust some odd wording, e.g. "mysterious challenges" (line 44), "harmoniously" (line 48) - I suggest to moderate the claims about the practical success of PINNs (see weaknesses for concrete examples).

Summary

In this paper, it is identified that PINNs can be adversely trained when gradients of each loss function exhibit a significant imbalance in their magnitudes and present a negative inner product value. To address these issues, the authors propose a novel optimization framework, Dual Cone Gradient Descent (DCGD), which adjusts the direction of the updated gradient to ensure it falls within a dual cone region. This region is defined as a set of vectors where the inner products with both the gradients of the PDE residual loss and the boundary loss are non-negative. Theoretically, the convergence properties of DCGD algorithms in a non-convex setting are analyzed. On a variety of benchmark equations, DCGD outperforms other optimization algorithms in terms of various evaluation metrics, achieves superior predictive accuracy and enhances the stability of training for failure modes of PINNs and complex PDEs.

Strengths

1. Theoretical analysis is provided, including the necessary and sufficient conditions under which the total gradient falls within the dual cone region, and the convergence rate. 2. Superior performance.

Weaknesses

1. It is better to provide the visualization of PINN predictions and compare with exact solutions, especially for the failure modes of PINNs. 2. Training timing is not reported.

Questions

Line 277, "Failure model" vs. "Failure modes"?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The proposed Dual Cone Gradient Descent (DCGD) optimization method for PINNs may still be trapped in local minima.

Summary

PINNs have become commonplace in both scientific computing and machine learning communities and have found widespread applications. There have been many modifications to various aspects of PINNs, such as the loss functions, initialisations, initial and boundary condition representations, learning algorithms and network architectures. A major problem with learning effective solutions through PINNs relative to other learning tasks is their difficulty in converging to the actual solution due to the complex nature of the underlying physics and equations involved and the corresponding loss landscape. The authors have provided a comprehensive survey of the previous works, making the unaddressed problems and goals very clear. A novel algorithmic framework based on convex optimization and geometric analysis is presented, which demonstrates superior performance on many tasks. The contributions of this paper are multifold, one being a new optimization algorithm while the other being a unified, open and flexible framework to adapt, modify and integrate the current technique with previous and future techniques. The paper is well written with enough information on the theoretical and mathematical underpinnings, experimental setup, implementation details, results and justifications.

Strengths

Although the concept of dual cone is well known in the optimization domain, specifically in the convex optimization area, the authors utilise and combine existing techniques from optimization theory in a novel way to develop a new algorithmic framework, which they refer to as DCGD. The authors propose the DCGD algorithm which involves picking a gradient from the Dual Cone region formed by the two losses (boundary loss and residual loss) of PINNs and also theoretically prove that it converges to a pareto-stationary point. Thus, along a new algorithm, theoretical guarantees for convergence are provided, which are very crucial for the goal of improving the training convergence of PINNs. With a suitable candidate being chosen as the new gradient from an explicitly defined space (called here as $G_t$), multiple variants of the DCGD algorithm can be created. The DCGD can also be combined with many existing optimization techniques, such as Adam Optimizer and Neural Tangent Kernel (NTK). This work introduces a fairly novel optimization technique which has huge potential for applications in PINNs and multi-objective optimization. The authors have provided strong justification for the generality of their work by showing that most previous MTL approaches are special cases of DCGD. The general, modular and extensible nature of the work paves the way to significantly accelerate further research in various allied areas. The supplementary material includes the datasets and code for all the experimental details mentioned in the paper and the appendices. I have glanced through the code and data files provided but have not personally verified the results by running the code. (might update here if I get to install and run the code provided)

Weaknesses

The authors have provided enough empirical and theoretical evidence for their proposed DCGD method, though since this work proposes a theoretical and algorithmic framework, albeit it would be notable to have a section on comparing the computational complexity and qualitative terms with existing algorithms. I would say a short section in Appendix D/E would go well. Since the problem at hand is of multi-objective optimization which overlaps with multi-task learning problems and PINNs are a natural suitable choice, it would be very helpful to see the results of this approach on PINOs (Physics Informed Neural Operators). There is no comparison (even qualitative) of DCGD on PINOs (There are PINOs which are trained only on residual loss, but here I'm specifically referring to PINOs which include both residual and boundary loss for being relevant to DCGD). Was this not considered or omitted, or is it expected to show improvement in performance, as PINNs have shown? (Personally, I believe it should show improvements as the authors have already demonstrated superior performance on PINNformer with DCGD) Presentations/Grammar/Typos (I do understand that the authors would be well aware of these and would rectify these in the pre-print or camera-ready versions. However, I would like to mention them here together to aid them. Some might have been missed; might update if needed) Equation 2.2: The weight of the boundary loss is mentioned as $\omega_u$ and also on line 80, whereas it is mentioned as $\omega_b$ (which seems more appropriate) on line 79 Section 4.3: (This is a nitpick and can be conveniently ignored) $g_t^c$ could be written on the same line as eq. 4.3 or as a newline as eq. 4.4 similar to how equation E.3 is written Section 5.2: Is the title intended to be “P(I)DEs” or “PDEs”? I believe this is just a typo Appendix A, line 769: Equations A.8 and A.9 could be mentioned here (as A.4 and A.5) and then referenced in the Proof of Proposition 4.6 section. I believe this will greatly enhance readability Appendix A: Proof of Theorem 4.2 line 757: The residual loss is repeated twice $\nabla\mathcal{L}_r(\theta_t)$ is repeated twice, it should be both boundary and residual loss, $\nabla\mathcal{L}_r(\theta_t)$ and $\nabla\mathcal{L}_b(\theta_t)$ Appendix A, line770: "One can derive that …" should we include the boundary loss term instead of residual loss as the derivation with residual loss is already shown in line769 i.e it should be $\langle g, \nabla L_b (\theta_t) \rangle \geq 0$ Appendix A, line 781: For the sake of notational consistency, could you replace "condition (2)" with "condition (ii)" Appendix A, line 785: (nitpick) The subscript t is omitted in the boundary loss and included in the residual loss ($\nabla\mathcal{L}_r(\theta_t)$ and $\nabla\mathcal{L}_b(\theta)$). I suggest the subscript "t" be included at both the places for notational consistency Appendix A, line 818, Proof of Corollary 4.7: 3. DCGD (Center): “vector” instead of “veoctr”

Questions

I have gone through the previous works mentioned in detail, and does seems like the problem of multi-objective optimization or even optimization, specifically in the case of PINNs, has not been looked at from the perspective of Dual Cones. Could the authors add some more works related to Dual Cone-based techniques to the previous/related works section? Typos and presentation issues are already covered in the weaknesses section; however, here, I would like to mention a few pointers where I would like to seek clarification. (This is more from a clarification perspective for me individually than a comment on the work, though I believe these clarifications will help the broader community) Appendix A, line 779, Proof of Theorem 4.5: In the derivation of line779, from the first step to the next, should there be an equality sign in place of the first inequality sign as we are just distributing the dot product over addition on the first term $\nabla \mathcal{L}(y+t(x-y))$? (i.e. the first of the 3 inequality signs should be an equality instead) Appendix A line 797: Is the expression $\left| \cos(\phi_t) - \pi \right| < \alpha$ is correct and as intended? The inequalities on lines 810 and 811 and the threshold criterion in the algorithms in Appendix E ($\pi - \alpha < \phi_t \leq \pi$) point otherwise. I believe there should not be the cosine of $\phi$; rather, it should just be the angle/phase difference $\phi$. I have a similar comment for line 984, as the same expression is used. Appendix C.5 line 925: Is the statement complete? Is it intentionally left this way?

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

I am appreciative of the authors who have mentioned (and also acknowledged in the checklist) a few limitations of the work in section 6 this is more from a future work perspective than from a limitations of the work presented as part of this paper. The limitations of the current work are not very evident from the present text. It would be further helpful if they could include a more detailed section on where this cannot be applied or other multi-objective scenarios where the DCGD optimization framework is not as effective as other previous techniques.