Investigating the Ability of PINNs To Solve Burgers' PDE Near Finite-Time BlowUp

Our bounds can be invoked on the approximate solution obtained from any kind of a PINN loss.

We thank you for your kind comments about our work. In the following, we address the key concern you have raised. > Can the author's theoretical framework be applicable to PINN variants, such as self-adaptive weight PINN, adaptive sampling, or adversarial training? Although our bounds in the theorems are computing errors in the $L^2$ norm, the proof can be applied to any surrogate solution found irrespective of the method used to obtain it (like the choice of norms in the loss function used) - in other words, our bounds will hold as long as the initial and boundary conditions are kept similar to ours for any method solving the Burgers' PDE and returning a function as a candidate solution. For eg. in self-adaptive weighting (https://arxiv.org/abs/2009.04544) they modify the loss function by adding specific weights to each residual and these weights are updated in each time step. In adaptive sampling (https://arxiv.org/abs/2210.00279), they choose a set of points in each step of training on the basis of ``failure probability'' at a few points in space. In both of these cases, the modifications do not affect the applicability of our bound since that only depends on being able to estimate the required boundary, initial conditions and bulk error computing integrals for the surrogate at the end of training - howsoever obtained.

Added new experiment details and literature reviews in Appendices C, D and E

We thank the reviewer for their detailed analysis of our paper and the kind comments. Firstly, we would like to point out that in the newly added Appendix C we have now given a side-by-side comparison between the true PDE solution and the neurally found approximate PDE solution for a sequence setups with 1D Burgers' PDE whose true solutions are increasingly getting close to being singular. Secondly, please see Appendix D where we have demonstrated how the run-time to solve 1D Burgers' PDE by the PINN method is barely affected by the proximity to the singularity of the true solution. Thirdly, kindly note the new Appendix E where we have reviewed that to the best of our knowledge existing results in the classical numerical analysis literature don't seem to be applicable to analyze PINN experiments with Burgers' PDE. In the following we shall address the key concerns you have raised. >it might not cover the spectrum of PDEs used in real-world situations To the best of our knowledge there is hardly any universal principle for how finite-time singularities appear in different PDEs. In our introduction, we have reviewed the evidence for such singularities in various famous models like the Frank-Kamenetskii PDE, 2D Boussinesq PDE, and the 3D Euler PDE. Each has its own peculiarities and mechanisms of blowing-up and we believe that each will require its own new analysis. We hope that our work could be a starting point for various such analyses in the future. >The paper could have delved deeper into comparing its method with other leading neural network solutions for PDEs. The theoretical framework proposed by us can be invoked on *any* surrogate solution found for the specified Burgers' PDE. The *applicability of our theoretical bounds does not depend on any specific method used for solving the PDEs* as long as the chosen numerical method gives a function approximation to the true PDE solution - as PINNs do, which is the primary method we focus on. >the study works under the assumption that we always know the main equations driving the physical system. In this work we focus on PINNs, which is a method of using deep-learning to solve PDEs in an unsupervised way - and by definition, this method is applicable only when the exact PDE is known. We posit that this is a common use case in the real world - and why PINNs are getting increasingly deployed for various uses. In cases where the underlying PDE is unknown but some input-output function samples (like samples of solution for different initial conditions) are available, operator methods such as DeepONets can be used. Formulating the problem of stability of operator learning in the face of finite-time blow-ups is an entirely different question (about learning in infinite dimensions) than what we focus on. >What are the generalization bounds for Burgers' PDE and how are they correlated to the neurally found surrogate solution? In usual ML parlance when we want to prove bounds on the generalization error for a setup, we are looking to bound the difference between the population risk and the empirical risk of a loss function. As we have pointed out in the review given in our Section 2, such bounds for PDE solving losses is available only for special kinds of linear PDE. Burgers' PDE being non-linear is not in the ambit of such existing proofs - and that is one among the many motivations for our work. Towards such a goal we posit that it could be an interesting direction of future exploration to try to understand the Rademacher complexity of the loss class considered here.

Added new experiment details and literature reviews in Appendices C, D and E

We thank the reviewer for their kind reading of our work and for appreciating the challenge that we have tried to solve. Firstly, we would like to draw your attention to the newly added Appendix E in the draft where we have reviewed multiple classical results in numerical analysis for Burgers' PDE to establish that to the best of our knowledge even for this smallest non-trivial case of finite-time blow-up, classical error bounds do not exist which can be invoked on PINN experiments with Burgers' PDE. Thus, our bounds and experiments in this work can be seen to be a first-of-its-kind result to address a challenging interface between deep-learning and non-linear PDEs. >Does the error bound (5) converge to zero in certain situations? Thanks for raising this interesting question! As you would note we have specified this exact question as an open question in our conclusion in the language of "stability" - which we have briefly defined in the footnote on page 6. For PDEs for which the ML method of solving it has a "stability" guarantee in this precise sense are the very cases where we can get a data-driven sufficient condition for the error of the approximation to be $0$. As we have pointed out in our discussion below Theorem 3.2, that this property of stability is not immediate for many of the ML losses used for PDE solving and one of our key observations (in Theorem 3.2) is that for Burger's PDE in one dimensions such a stable bound is possible despite allowing for the difficult solution of a finite-time blow-up. >Although it is interesting that certain correlations between RHS and LHS of the inequalities are observed, the magnitudes of them are very different. So, I am not sure whether these results are meaningful or not. As we have emphasized in the literature review done at the beginning of Section $4$, getting non-vacuous generalization bounds for deep-learning is an extremely open problem. The objective of our research here is not to try to solve this question which is open for even much more basic setups with neural nets. The crux of our work - in particular our experiments - is to establish that the theoretical bounds we prove are not only applicable to this involved setup of nets solving PDEs but that they also evaluate to values that are highly correlated to the functional distance of the found surrogate from the true PDE solution. We posit that it is somewhat of a surprise that this correlation between the bound and the truth is strong (approaching nearly $1$ at times) despite the vacuity of the bound and despite the proximity to singularities - which is a unique feature of our PDE under consideration.

Our use of the term "generalization error" is consistent with foundational papers in PINN theory

Thanks for your comments. In usual ML, the word "generalization bound" typically refers to a bound on the gap between the population risk and the empirical risk and this can often be shown to have easy sufficient conditions of approaching $0$ by methods such as Rademacher complexity. *But in our theorems here, the LHS is not this notion of generalization gap* but a measure of the distance between the surrogate function and the true PDE solution. In the context of PINNs this quantity has been called as the "generalization error" right from the first theory papers in PINNs like the canonical result from 2022, Theorem 2.6 here, https://doi.org/10.1093/imanum/drab093 - which laid the foundations of the subject. Please note that the authors there too called this gap from the true PDE solution as "generalization error" But kindly note that the bounds like in the foundational papers use an unconventional notion of training error (which carefully weights each collacation point) which is not implemented in PINN experiments these days but our bounds don't have such a condition - and thus we take a step forward compared to them and build insights closer to experimental setups. So, we posit that our use of the word generalization error for this quantity is consistent with the foundational papers in the theory of PINNs while we take a step forward in many other ways. Maybe we kindly request the reviewer to take cognizance of this fact.

Added new experiment details and literature reviews in Appendices C, D and E

We thank the reviewer for their critical reading of our work. May we request you to reconsider your scores in light of the explanations that we provide below for the $5$ weaknesses you have pointed out. > As the equation approaches blow-up, the $L^\infty$-norm of the gradient approaches infinity. The bound thus does not provide meaningful information in the vicinity of the blowup. We suggest reading our bounds not at the blow-up point but in its vicinity i.e when $\delta \in [0,1)$ for say Theorem $3.2$. At any such $\delta$ the bound is finite and that's when we do our experiments. The meaning we extract from these bounds is as displayed in Figures $1$ and $2$ where we show that despite the bounds being computed arbitrarily close to a blow-up they maintain correlation with the distance of the neurally found surrogate from the true solution. To the best of our knowledge, except this work, currently there are no other theorems available off-the-shelf which can be used to get any bound at all on the population risk of a neurally found surrogate solving for Burgers' PDE near a finite-time blow-up. We would like our work to be seen as a critical first step towards understanding the training of neural nets to solve PDEs near a blow-up. >[The $(L_2,L_2,L_2,L_2)$-stability claim] is misleading We would like to draw your attention to how stability has been defined in the footnote of page $6$. This definition is inspired from an earlier work, https://doi.org/10.48550/arXiv.2206.02016. As you would note that this definition is about an order estimate - that if the population risk of the PINN loss is ${\cal O}(\epsilon)$ then it should be a sufficient condition to ensure that the distance of the surrogate from the true PDE solution is also ${\cal O}(\epsilon)$. We note that in general for arbitrary PDE, it is not guaranteed (and may even be provably impossible) for the PINN loss to have this property of stability. And this is the special phenomenon that happens in our Theorem $3.2$. Thus even when the true solution is unknown, a PINN loss could be proven to be stable if the bound on the true risk can be shown to depend on the true solution such that the dependencies can be absorbed into the constants kept implicit in the ${\cal O}$ estimate. >"generalization error" term for Theorem 3.1 is misleading. These type of stability estimates (of the operator mapping right hand side and initial condition to the solution) are standard tools in the theory of partial differential equations, making this seem more like a re-branding. It would strengthen the paper if the authors would discuss related results in the PDE literature. We would like to emphasize that the key message of the theorems like what we have proven here is to able to estimate the proximity of a surrogate solution from the true PDE solution in terms of errors made by the solution in satisfying the PDE in the interior of the domain considered and the errors made in satisfying the boundary and the initial conditions. To the best of our knowledge, there are scant results in conventional PDE literature which give such bounds. Kindly see the newly added Appendix E where we have reviewed various classical literature to establish that to the best of our knowledge existing results in numerical analysis cannot be deployed to understand PINN training - as is the target here. In particular, specifically for the $0$ viscosity Burgers' PDE we can only find results like this, https://doi.org/10.1090/S0025-5718-1987-0906180-5, where the theory does not give a bound on the distance from the true solution of the PDE to that found by the finite element method. More generally, in works such as Corollary 3.5 in https://doi.org/10.1137/0728048 the authors consider a weak solution of the $\varepsilon$-viscosity regularized Burgers' PDE and derive bounds on the local $L^p$-distance between the weak solution and the actual solution. There is no obvious way to apply these bounds for a PINN solution since the trained net has no guarantee to be satisfying the conditions required of the surrogate here. For results like Theorem 2.1 in https://doi.org/10.1137/0729087, we observe that these too don't have an obvious way for the bounds to be applied for PINN experiments because they need stringent conditions (like satisfying the conservativeness property) to be true for the approximant, for the bounds to apply and there is no natural way to know if the neural surrogate satisfies these conditions. Also neither of the above cited classical bounds are tailored to any compact domain and hence there is no boundary condition error that is getting tracked there as in our Theorem $3.2$. In contrast to such literature, even in the context of existing PINN guarantees, we posit that our theorems are a step towards novel and useful guarantees.