Physics-Informed Diffusion Models

Clarifying the setting of PIDMs

We thank the reviewer for their critical evaluation of our method and for providing a list of references on leveraging diffusion models to address sampling problems. **Clarification of core contribution** We would strongly like to clarify that mere sampling from an unnormalized target distribution is *not the setting in which we operate*. Instead, we address the common setting that involves leveraging the expressivity of diffusion models on available datasets in the natural and engineering sciences. For only a selection of recent works that operate in this exact setting, we have provided the references *Xie et al., 2021; Buehler, 2022; Bastek & Kochmann, 2023; Vlassis & Sun, 2023; Sardar et al., 2023; Li et al., 2023; Lienen et al., 2023* in the manuscript. Unlike the references provided by the reviewer, our setting assumes the availability of data, which fundamentally distinguishes it from the sampling-focused approaches highlighted by the reviewer. (Moreover, Sanokowski et al. operate in a combinatorial space, whereas we consider continuous solution fields, raising questions about the applicability of their results to our setting.) Concretely, our model is not designed to learn all possible samples that fulfill a constraint but rather to augment data-driven diffusion models by grounding them in the known physical principles that govern the available data. Adopting a purely residual-based approach, closer to the references provided, would require the diffusion model to primarily *function as a numerical solver and thus compete with decades of research into highly specialized solvers for the corresponding PDEs*. While this setup is possible, we do not believe this to be a realistic or practical use case, and note that this might also explain why none of the provided references address PDE-based systems. Additionally, we emphasize that all theoretical insights are derived under the assumption of this specific (data-driven) setting. In Appendix A.1, we demonstrate how our augmented loss continues to recover the data distribution, and in Appendix A.3, we formulate a simplified loss based on the assumption that samples from the data distribution are available. This approach aligns with prior work aimed at grounding diffusion models in physical principles in scientific applications, such as [1-4], to which we have extensively compared our framework. We hope the reviewer acknowledges this critical distinction and reconsiders their negative evaluation. **Question 1:** As highlighted in Section 3.3, a primary challenge in diffusion models is that directly sampling $\boldsymbol{x}_0$ is computationally infeasible. A significant contribution of our work is the development of two frameworks—both outperforming the state of the art in residual error reduction—that circumvent this complexity. These frameworks either use the (straightforwardly available) mean estimate or an accelerated sample estimate that requires only one additional forward pass. We provide theoretical arguments demonstrating the consistency of the latter approach, which is experimentally validated in Figure 2c. Since the subsequent steps we introduce remain independent of using either the mean or the sample estimate, we simply introduced $\boldsymbol{x}_0^*$ as a notation that may relate to both estimates. We acknowledge the remark from the reviewer that this may be unclear and have clarified this in the revised manuscript. **Question 2:** We have provided studies on the effect of removing the data-based loss term in Appendix A.6.1 (Figure 5, 6; scenario (iii)). In this case, the distribution simply collapses to a random point on the constraint manifold and resembles the training of a classical PINN. We observed a similar behavior in the fluid flow case study but did not include it, as, again, this is not the scenario we envision with PIDMs. [1] Dule Shu, Zijie Li, and Amir Barati Farimani. A physics-informed diffusion model for high-fidelity flow field reconstruction. Journal of Computational Physics, 478:111972, 2023. [2] Christian Jacobsen, Yilin Zhuang, and Karthik Duraisamy. CoCoGen: Physically-Consistent and Conditioned Score-based Generative Models for Forward and Inverse Problems. [3] François Mazé and Faez Ahmed. Diffusion Models Beat GANs on Topology Optimization. Proceedings of the 37th AAAI Conference on Artificial Intelligence, AAAI 2023, 37:9108–9116, 2023. [4] Giorgio Giannone, Akash Srivastava, Ole Winther, and Faez Ahmed. Aligning Optimization Trajectories with Diffusion Models for Constrained Design Generation. Advances in Neural Information Processing Systems, 2023.

Follow-up clarification on answer to rebuttal

We would first like to thank the reviewer for taking the time to go through our response, providing timely feedback, and engaging in this interesting discussion. We appreciate the opportunity to clarify the contributions and novelty of our work. We agree with the reviewer that Equations 7–9, when viewed on their own, can be understood as sampling from an unnormalized distribution constructed by evaluating $q_{\boldsymbol{\mathcal{R}}}$ over $\boldsymbol{x}_0$. This approach aligns with the setting introduced by Rixner & Kourotsekalis (2021), whom we have cited in the relevant parts of our paper. As we acknowledge by referencing them, this is not our key contribution. Again, we emphasize that without considering data, we compete against highly optimized numerical solvers designed for the specific choices of PDE. Our key contribution follows afterward, where we augment the common data-driven setup of diffusion models to improve adherence to the set of PDEs under consideration. Unlike the sampling problems as given by the reviewer, we train the model *not* by sampling from a simple prior; instead, we query our model on noisy latents obtained from the data points, changing training dynamics as the model operates on latents that are aligned with the true distribution. More importantly, a key contribution of our framework lies in identifying **simple, but effective surrogates to obtain the clean signal at any point in the diffusion trajectory**, for which we have provided both the so-called **mean** and **sample estimation**. Both methods successfully outperform prior SOTA in terms of residual fulfilment. By contrast, all references of the reviewer require sequentially sampling over the whole diffusion trajectory to obtain a sample based on which the energy or unnormalized density is evaluated. This increases training time by orders of magnitude and also has significantly larger memory requirements than the standard (data-driven) score matching techniques (to which we count our approach). To our knowledge - and we thank the reviewer for pointing out this interesting connection we were not aware of - the only partial exception is indeed the 'annealed noise distribution' by Sanokowski et al. Nevertheless, **we would like to clarify that this setup is fundamentally different to our equation 12**. As mentioned in the prior paragraph, a key insight is that **we evaluate our residual on the mean or sample estimate** of the clean signal, while Sanokowski et al. evaluate it on the **latent** (i.e., $X_t$ ($\boldsymbol{x}_t$ in our notation), see bottom right of page 4 in Sanokowski et al.). In fact, we have pointed out in Appendix A.1 that evaluating the residual on such latents is not consistent: Adding noise to the solution field drastically increases the residual error due to its high sensitivity to higher-order derivatives of the solution fields, rendering this loss ineffective. Early investigations confirmed this issue (which we are happy to reproduce if it is of interest to the reviewer). Instead, we think it could be of interest to apply our estimation techniques for $\boldsymbol{x}_0$ to their setting. Regarding topology optimization, while the material distribution is indeed a continuous relaxation of a binary design space, we highlight that the physical fields (i.e., displacements), which dictate the performance of the design and are part of the model predictions, are not discrete. Also, we clarify that we have in fact considered a form of entropy regularization by considering Gaussian noise as data, as briefly investigated in Figure 5, 6, case (iv), and observe that this encourages the model to obtain diverse results that still respect the constraint. Lastly, we point out that we have presented two studies in relevant fluid and structural mechanical settings (e.g., [1] solely focuses on the latter setting and was published in a similar top-tier conference), besides a toy study (which is similar to the experiments shown in some of the references the reviewer has provided). While we agree that additional studies would further showcase the applicability of PIDMs, we disagree with the notion of a 'limited experimental validation'. Once again, we would like to thank the reviewer for pointing out the interesting connections. We have included the suggested references in the appropriate section (3.1) in our revised manuscript. At the same time, we would greatly appreciate it if the reviewer could reevaluate their assessment of the 'substantial methodological overlap' given the clear theoretical and technical distinctions listed above. [1] Giorgio Giannone, Akash Srivastava, Ole Winther, and Faez Ahmed. Aligning Optimization Trajectories with Diffusion Models for Constrained Design Generation. Advances in Neural Information Processing Systems, 2023.

Raised score

I appreciate the authors' clarifications regarding their methodology. Upon careful reconsideration, I acknowledge that I previously overlooked their approach of rewriting the expectation over model samples to an expectation over data samples by assuming uniform importance weights. In light of this understanding, I am revising my score to 5. Since the authors are exclusively working with expectations over data samples, I believe their method is working in a similar setting as the "Guidance methods" of Diffusion models, such as the work by Yu et al. ("Freedom: Training-free energy-guided conditional diffusion model"). To strengthen the paper, I recommend conducting a comparison with existing guidance-based methods. Additionally, I suggest the authors include a reference to Sanokowski et al., due to the notable similarity between Equation 12 in this paper and the annealed noise distribution presented in their work. Yu, Jiwen, et al. "Freedom: Training-free energy-guided conditional diffusion model." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Answer to request for feedback

I agree with the authors that their initial contribution rating of 1 does not adequately reflect their work. Therefore, I have increased the contribution score to 2. However, I will maintain my current overall score of 5 because I consider the scope of experiments and the contribution to be small. Finally, I noticed that on lines 29, 379, and 397, question marks (?) have been inserted where proper citations should appear. These sections require appropriate academic references to be added.

Answer to reviewer

We appreciate the reviewer's reconsideration, though we respectfully disagree on the small scope of the experiments as we have indeed provided 'a comparison with existing guidance-based methods', as requested by the reviewer and evident in the supplementary code, and other frameworks operating in this setting. We thank the reviewer for identifying the lines that were not properly compiled and will ensure these issues are resolved in the final version upon acceptance.

Clarifying model performance and benefit of simple loss

We thank the reviewer for their valuable feedback and thoughtful questions. Below, we address each point in detail. **Weaknesses:** Allow us to first clarify on the metrics: In the Darcy flow example, the PIDM with sample estimate *outperforms all other variants*, including the standard diffusion model and prior SOTA, on the two relevant metrics: residual error and data loss (see Fig. 2). Notably, the PIDM does not only reduce the residual by order of magnitudes but is also more robust to overfitting by matching the underlying physical ground truth. For the topology optimization study, we fully acknowledge that not all metrics are best for the PIDM. However, the metrics must be interpreted more nuanced: For the compliance and volume fraction error, there will always be a trade-off as adding more material (i.e., increasing the volume error) will always lead to a stiffer structure (i.e., reduced compliance error). Thus, both of these measures have to be taken jointly into account. Depending on what is valued more, the *PIDM performs similarly if not better compared to frameworks specifically designed for this task*, which require auxiliary models and potentially additional 'noised' data [1,2], thus drastically complicating the framework. In contrast, our approach is intuitive and simple to implement, which, while not explicitly listed in the table, underscores the framework’s generality and ease of implementation. This simplicity highlights its potential for broader applicability across various domains. The reviewer raises the good point that the integration of a residual loss term is 'straightforward', which we in fact fully agree with and would actually like to clarify on this: It was specifically our goal to come up with a method that is *(i) general applicable for any physical residuals, (ii) effective in improving the physical accuracy, and (iii) straightforward to implement*. Alternatives such as [2] with similar objectives require, as mentioned before, auxiliary datasets and models, and we see it as an advantage that our method reduces to a simple extension of the loss. *The fact that we arrive at this simple form is, and we want to emphasize this, the exact intention of our derivations*, which are fully motivated from first principles via virtual observables. In fact, we have shown all relevant approximations and simplifications that must be introduced to exactly arrive at the 'simple' loss given in Eq. 13. Furthermore, key contributions are the two alternative mean and sample estimates to efficiently evaluate Eq. 13, which is a challenge specific to diffusion models. We would also like to point out that this straightforward integration of physics into this setting was, to the best of our knowledge, not obtained by prior work, though informing diffusion models on physics has been the theme of various contributions, as listed in the manuscript. We suspect that the understanding of the general setting of introducing deterministic constraints into probabilistic diffusion models, as well as specific challenges such as the computationally intensive, iterative sampling, are the prime reasons for this gap in the literature that we fill here.

Clarifying notation

We thank the reviewer for acknowledging our 'compelling' contribution that can benefit the machine learning community in the scientific setting. We also thank them for pointing out difficulties in following the notation introduced in our paper. We have extended the explanations in our revised manuscript (highlighted in red) and clarified our contributions in the abstract. We have also included the additional Appendix A.9 to clarify the assembly of the residual. The variable $\boldsymbol{\xi}$ is always reserved for spatial coordinates that are relevant in the assembly of the residual, while $\boldsymbol{x}_0$ always refers to a data point. This is in our context typically an image that contains the relevant solution field values $\boldsymbol{u}$ at a collection of discretized spatial coordinates $\boldsymbol{\xi}$. We hope this clarifies potential ambiguities and hope to have addressed the reviewers concerns. If further ambiguities prevail, we are more than happy to address them.

Thanking the reviewer

We thank the reviewer for their constructive feedback and for acknowledging the improvements in our revised manuscript. We appreciate their recognition of our contributions and the updated evaluation. Upon acceptance, we will move the definition of $\mathcal{R}$ to the main article.

Clarifying distinction from Rixner & Koutsourelakis, 2021

We thank the reviewer for their thoughtful review and feedback. We sincerely appreciate their recognition of our paper’s originality, quality, clarity, and significance, as well as the potential of our method to enhance generative modeling for scientific applications. We understand the concern regarding the clarity of our contribution, particularly in relation to prior work on virtual observables [1], which we have duly acknowledged in our manuscript. As a starting point, we adopt the virtual observable concept as presented in Eq. 8 of [1] (corresponding to Eq. 7 in our manuscript). *However, the remainder of our manuscript differs significantly in all subsequent derivations (and implementation).* Unlike [1], we do not compute an ELBO approximation over the joint distribution. Instead, we treat the physical likelihood as an auxiliary training objective, analogous to the original framework of PINNs [2]. For this auxiliary physical likelihood, we avoid approximating the ELBO and instead evaluate it exactly when sampling from the model. The theoretical insights underlying this auxiliary loss term, including its exact sampling mechanism, are detailed in Appendix A.1 and A.3. The key challenge in the context of diffusion models lies in the computational infeasibility of exact sampling, requiring accurate and efficient alternatives. Addressing this challenge forms a major contribution of our work, which is completely irrelevant in [1]. Specifically, we propose two novel approaches for efficient sampling—both surpassing prior state-of-the-art methods in terms of residual error that may be of independent interest: **Mean Estimation**: This method leverages the estimated mean of the clean signal given a noisy input, avoiding additional forward passes. However, it introduces an inconsistency due to the Jensen gap, as the residual evaluated on the mean does not necessarily equal the mean of the residuals. **Sample Estimation**: In this approach, we obtain a sample estimate of the clean signal through accelerated sampling schemes, particularly deterministic Denoising Diffusion Implicit Models (DDIM) [3]. While requiring an additional forward pass, this method significantly reduces inconsistencies between the data and residual losses. To the best of our knowledge, this approach has not previously been integrated into the training process (as opposed to inference). We provide theoretical justifications for the superior performance of sample estimation over mean estimation in our context, supported by clear numerical evidence (see Figure 2 in the manuscript). To summarize -- while we also start with the virtual observable as introduced in Eq. 8 of [1], our subsequent setting and contributions are fundamentally distinct, particularly in addressing the unique challenges of diffusion models. Unlike prior work, we do offer a fully first-principle based derivation of the integration of physics into diffusion models, do highlight all relevant simplifications and potentially introduced biases, and demonstrate SOTA performance on most relevant metrics in two relevant settings. We hope these clarifications address the reviewer's concerns and highlights the originality and significance of our work and would appreciate a reconsideration of the score. [1] Maximilian Rixner and Phaedon Stelios Koutsourelakis. A probabilistic generative model for semisupervised training of coarse-grained surrogates and enforcing physical constraints through virtual observables. Journal of Computational Physics, 434, 2021. [2] M. Raissi, P. Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686–707, 2019. [3] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. International Conference on Learning Representations, 2021.