Physics-Informed Variational State-Space Gaussian Processes

Summary

The paper introduces a method for training spatio-temporal Gaussian processes which incorporate physics constraints including satisfying the governing equations at a number of collocation points and satisfying curl / divergence free constraints. Their approach scales both linearly in time and space by leveraging a number of approximations.

Strengths

- While many of the individual components (such as incorporating curl / divergence free constraints and satisfying governing equations at a number of collocation points) have been done before, I believe the combination of ideas here is novel. - Showing how to achieve linear complexity in the space and time dimension is valuable. - Showing how your approach recovers some prior works as a special case from a more general perspective is also valuable. - The bevy of test cases demonstrating the efficacy of your approach on both synthetic and real-world problems convincingly demonstrate the advantages offered by your approach.

Weaknesses

- The problem statement is poorly motivated in my perspective. It would be helpful to be more specific about what you hope to achieve with hybrid modeling. For example, is your goal to improve the predictive accuracy of mechanistic models by incorporating data? What computational efficiency do you hope to improve? i.e. do you want to reduce the amount of data needed to train physics surrogates? Do you argue your approach is more efficient than solving the mechanistic model using traditional methods? etc. While I understand space is limited, being more specific about the problem statement will also help guide more targeted numerical studies in future works. - I think you need slightly more of a discussion on virtual observations (see questions below). When virtual observations are competing with actual data it would be helpful to discuss in detail how you think about $\epsilon_C$ and choosing the number of collocation points. - Since you state that one of the goals of your approach is to be useful in uncertainty quantification, the numerical studies would have been strengthened by comparing more than just RMSE. For example, comparing the continuous ranked probability score (CRPS) would have given an indication of how well your approach is estimating uncertainty. - Minor: - L94 space cases -> special cases?

Questions

- As I understand it, you can always choose to satisfy the differential equations at enough collocation points such that you overwhelm the data. For example, say I have very limited data so I choose the number of collocation points to be much greater than the number of data points. Clearly, if I use too many collocation points, my model will ignore the data and just follow the differential equations. How do you think about this perspective in the context of uncertainty quantification? - As a follow up to this previous point, what are the implications of assuming that $0_n^{(C)} = g(F_n) + \epsilon_C$? Is it correct to think about this as implicitly assuming that the PDE is stochastic? - Are there situations where you believe the Gaussian process assumption could be limiting?

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

2

Limitations

I think the authors have done a good job of identifying some potential limitations. I think the paper would greatly benefit from a more in depth discussion on using collocation points to enforce governing equation constraints and the limitations this perspective brings.

Summary

The paper introduces a formulation for state-space GP which aims to capture the behaviour of PDE-based systems. The paper suggests multiple techniques to speed up the GP inference, including through decomposed kernel, variational methods on natural parameterisation of distribution, use of inducing points

Strengths

The paper provides good motivation and use cases for physics-informed GPs, and the need for scaling the inference process. The paper also provides adequate results and demonstrations of the GP for prediction and uncertainty quantifications, which well motivates the benefits of GPs over other solver types.

Weaknesses

- The paper may be a bit hard to follow if one is not familiar with GPs already. For example, it may be difficult to immediately see how the PDE or the BV are incorporated into the probability distribution during the inference stage. - It may also be arguable that some parts of the paper can be seen as just application of standard sparse GP techniques and variational methods into the context of physics-informed GPs.

Questions

- Are there any intuition as to why the decomposition of the kernel is done between the spatial and the temporal variables? Is it for computational reasons or can it be linked to properties of physical phenomena? - Does (or can) the method perform in the case that the full form of the PDE is unknown? Or is the paper mostly focused on cases of just generating the PDE solution? - In the formulation given by Eq 6, the PDE and BC residuals are assumed to arise from a Gaussian distribution. How can they be related to the physical realisation of the PDE, or how can they be interpreted properly as a probability distribution? How realistic is this assumption? - Are there any additional assumptions in the form of g(f) should take for this method to work?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are adequately mentioned.

Summary

The paper introduces a novel approach for solving partial differential equations (PDEs) using a Gaussian process prior. It leverages established methods for approximate inference and provides a cohesive framework that unifies existing results in probabilistic differential equation solving.

Strengths

The paper introduces a novel approach for solving partial differential equations (PDEs) using a Gaussian process prior. It leverages established methods for approximate inference and provides a cohesive framework that unifies existing results in probabilistic differential equation solving.

Weaknesses

The paper’s presentation is challenging to follow. Despite my familiarity with continuous-time filtering, smoothing, Gaussian process (GP) literature, and continuous-time stochastic optimal control (where some of these partial differential equations (PDEs) appear), I struggle to determine the paper’s solidity. Numerous mathematical inaccuracies and inconsistencies hinder comprehension. For instance, the abrupt switch between notation for space variables (e.g., $s$ and $t$) and the combined space-time variable ($x$) lacks clear definition. While I recognize that machine learning sometimes necessitates complex notation, in this case, it impedes understanding. The main model section exacerbates the issue. Equations 5 and 6 remain elusive; the dimensions of $Q$ and $P$ are unspecified. The definition of $F_n$ confuses me, as it prevents a sensible matrix product computation. Additionally, the collocation point definition remains unclear. Even in Example 3.1, the meaning of $g_k$ remains unknown to me. Consequently, assessing the paper’s solidity proves challenging. In my view, this paper requires substantial revision before acceptance. Although I appreciate the effort invested, from a reviewer’s standpoint, the current presentation poses significant hurdles.

Questions

- In the background section please define N and F. And the relations between al the other variables N_t, N_s x_{t,s}, y_{t,s}, d, etc. I find this very confusing – is there also a clash in notation? - In Eq (3) can you elaborate if $\mathbf f$ is multidimensional and how does it relate to $f$. This notation is again used in line 78. - In line 66 is this definition correct? Somehow I have a hard time coming up with the correct dimensions. - In line 70 $\bar{\mathbf f} $ is a time depdent function. But in line 78 it is a space time depdent function. Could you elaborate on this? - In Line 87 is the non-linear differential operator only dependent on space derivatives, or can the also be higher order time derivatives? - Eq (5): $\bar f_q(\mathbf X_n)$ should be a vector of dimensionality D, but in line 98 W is of dimensionality $PD \times QD$. Can you elaborate on this. - In eq(7) can you elaborate what $g(\mathbf F_n)$ is? Is it an element-wise operation for each element in $\mathbf F_n$? - In line 265-266 can you please elaborate what $z$ is?

Rating

4

Confidence

2

Soundness

1

Presentation

1

Contribution

2

Limitations

-

Summary

In this paper, the authors present a physics-informed Gaussian process based approach to learn the solution of ODE and PDE systems. In particular, they address the challenge of the cubic computational complexity with respect to the number of spatial observations. It is shown that multiple state-of-the-art approaches can be recovered as special cases of the presented approach, i.e., it is a unifying framework. With additional approximation techniques, the cubic spatial computational cost is reduced to linear. In multiple simulations, the proposed method achieves similar or better results compared to SOTA methods but with a significant reduced computational time.

Strengths

- The proposed method demonstrates a significant improvement of the computational complexity with respect to the number of spatial observations compared to SOTA methods. - The beauty and originality of the proposed model is its unifying property, including existing approaches such as HelmholzGP and AUTOIP. - The techniques and ideas in the paper are clearly presented. I really enjoyed reading the paper.

Weaknesses

- There are some typos such as line 103 “develop sate space algorithms for”, line 167 “complexity of O(N (N_s” (missing t), and eq (17) Nt instead of N_t. Furthermore, I think that line 257 “RMSE of all models increases as the number of collocation points increases” make no sense as the RMSE is decreasing. Please check the paper carefully. - In the experimental section, some phenomena are rarely discussed. For instance, the RMSE of PHYSS-GP is increasing with 1000 collocation points.

Questions

- I assume that there exist PINN based approaches with some form of uncertainty quantification. It would be interesting to compare the GP-based method to NN-based approaches (not necessarily for this paper but in general). Do you have any thoughts on that? - Is it possible to use the same approximations in AUTOIP to reduce the cubic computational cost? If so, is there still a benefit of the proposed method?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

The submission discusses conditioning spatiotemporal Gaussian processes using differential equation constraints and observational data. Specifically, the temporal component is handled by a Markovian prior (to achieve linear complexity), and the spatial component is dealt with by variational methods. As such, the proposed algorithm extends the work by Hamelijnck et al. [18] to differential-equation constraints (and to handle spatial mini-batching); or, from the opposite perspective, it implements a version of a probabilistic numerical solver for spatiotemporal PDEs with variational methods (in space). The resulting algorithm is evaluated on a damped pendulum ODE, a curl-free magnetic strength field, a one-dimensional reaction-diffusion PDE, and an ocean-current problem.

Strengths

All components are technically sound, and the submission fits into recent literature on dynamical systems, Gaussian processes, and physics-informed machine learning. Overall, I think this is a nice paper, and despite a few minor weaknesses (see "Weaknesses"), I recommend acceptance.

Weaknesses

I identify two weaknesses: 1. The proposed algorithm's scientific novelty as a combination of known techniques is limited. Hamelijnck et al. essentially cover spatiotemporal variational inference with a Markovian prior in the temporal dimension. Conditioning Gaussian processes on differential equation constraints via collocation has become standard practice in recent years. Berlingheri et al. developed the curl- and divergence-free spatial priors. The combination of these techniques is novel, but the increment to the existing literature is relatively small. 2. The clarity of the submission's relation to existing approaches would improve by discussing more articles related to probabilistic numerical methods. Currently, the manuscript cites the PMM by Cockayne et al., the ODE (initial value problem) solvers by Schober et al., Krämer et al., and Tronarp et al., and the GP-PDE paper by Pförtner et al. (all references are in the submission). Beyond those papers, Krämer et al. (below) describe solving time-dependent, nonlinear PDEs with collocation and Markovian priors; Schmidt et al. (below) combine nonlinear collocation constraints with observational data (for ODEs); and Krämer and Hennig (below) solve boundary value problems with collocation. There are other related papers, but I believe the former articles should be discussed in the manuscript. Further, the literature on statistical finite element methods is related and should be acknowledged: see Duffin et al. (below). The first weakness is more significant than the second one. I expect that the second weakness will be relatively straightforward to resolve. In any case, I believe the strengths outweigh the weaknesses and lean towards recommending acceptance. **References:** Schmidt, Jonathan, Nicholas Krämer, and Philipp Hennig. "A probabilistic state space model for joint inference from differential equations and data." Advances in Neural Information Processing Systems 34 (2021): 12374-12385. Krämer, Nicholas, Jonathan Schmidt, and Philipp Hennig. "Probabilistic numerical method of lines for time-dependent partial differential equations." International Conference on Artificial Intelligence and Statistics. PMLR, 2022. Krämer, Nicholas, and Philipp Hennig. "Linear-time probabilistic solution of boundary value problems." Advances in Neural Information Processing Systems 34 (2021): 11160-11171. Duffin, Connor, et al. "Statistical finite elements for misspecified models." Proceedings of the National Academy of Sciences 118.2 (2021): e2015006118.

Questions

None

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

Discussed.