Summary
The authors propose incorporating derivative information into the training of DeepONets to improve predictive accuracy on PDE problems. Instead using an encoder-decoder neural operator architecture as in prior work, the authors use the DeepONet architecture as well as incorporate spatial derivative information on top of functional derivatives previously used. The authors provide numerical comparisons with other models using both types of derivative training as well as results for two types of input space dimensionality reduction methods.
Strengths
- The paper is very well written, well structured, and with the right level of detail and understanding to describe the setup in the preliminary section without lingering too long. The authors clearly have a deep understanding of all components of this method and convey the steps well in the space given.
- The appendix is thorough, high quality, and provides relevant details to better comprehend and reproduce the results in the main text. I found the visualizations in section B.6 particularly compelling for the low data case with vanilla FNO and DeepONet failing compared to the gradient enhanced models.
- The improvement in dm prediction for control is an important but overlooked metric for neural operators which primary focus on the output solution accuracy despite the gradient information being needed in real-world problems. The authors may want to consider the existing real-world example of DeepONets for optimization in [1], shown in Figure 7 to require gradient information of the objective w.r.t to the input parameters to optimize aerodynamic shapes which could benefit from this methodology which is a strength over accuracy improvement alone.
[1] Shukla, Khemraj, et al. "Deep neural operators as accurate surrogates for shape optimization." Engineering Applications of Artificial Intelligence 129 (2024): 107615.
Weaknesses
- The cost of generating the dm and dx labels with which to incorporate into training was 2-3 times greater than the cost of solving the PDE itself given Table 1. Of course, as with all neural operators the question is the tradeoff between offline data generation and training versus the benefit of quick online predictions so this cost could be mitigated given the accuracy benefit it provides. Nevertheless, its quite the increase and one may wonder if its worth doing at that point compared to just using the high fidelity FEM solver for N number of problems. Unless N is high enough, the tradeoff looks less appealing here.
- I would like to see the authors rephrase their findings in the context of the results in B.4 which show that the dx information provides minimal to no benefit compared to the dm training, and it some cases it even makes the accuracy worse than dm alone. This comparison of dx and dm alone should be stated in the main text as it appears to be critical information. Given that beside the solution accuracy, the dm accuracy is vastly more important that the dx accuracy to perform PDE-constrained optimization problems, it begs the question why dx regularization is done at all here. This then begs the question how novel this is compared to DINO which already trained on dm information. The authors claim Sobolev learning is novel on top of the DINO results but it does not appear impactful. The authors need to address this.
- Following onto the prior point, I am on the edge regarding impact and novelty here. That being said I still think the work is informative and of high enough quality to publish, albeit without as strong of an impact due to the main benefits shown to be in line with the existing method DINO. Looking at DE-DeepONet vs. DINO in Figure 1, there is not much difference, and in Table 2 DINO is shown to be much faster. I would consider changing my rating if this was adequately addressed.
Questions
- Could the authors please include total wall-clock time to Figure 2 in addition to per epoch. This would be helpful in comparing to the data cost in Figure 1 which is a total. Additionally, the authors may want to consider including a row for end-to-end data and training time such that the DeepONet and FNO include the PDE solution generation time and DINO and DE-DeepONet include the data for all three loss terms. This way the reader can very easily compare the total cost associated with making predictions with a DeepONet versus DE-DeepONet. I think this would benefit the manuscript since it mitigates the cost of the dm dx label generation.
- How can the epochs only be 1,000 in B.3? In the original DeepONet paper 50,000 – 500,000 iterations were used. Are all models converged to fairly compare them? What do the convergence plots look like? It would be helpful to see them.
- Is the vanilla DeepONet and FNO also trained with the dimensionality reduction technique on the input, and if so, which one? Figure 1 shows KLE and ASM for DE-DeepONet, and DINO is stated to use ASM, but what about the baseline models. How is it a fair comparison if the inputs are not the same?
- The usage of CNN for the trunk and ResNet for the branch should not be hidden in the appendix. The vanilla DeepONet uses a fully-connected NN for both the trunk and branch and that would be assumed here, please mention it in 3.1. Additionally, what justification is there for those choices? The CNN for the trunk makes sense to construct the basis but I’m curious about the ResNet for m?
- The authors should consider the following papers [2,3]. In [3] the authors train DeepONets the physics-informed (dataless) way using gradient information to obey the governing equation. In [2], the authors train a PINN with gradient-enhanced information which could also be done with the previously mentioned PI-DeepONet in [3]. How might DE-DeepONet methodology be incorporated into these models and in what ways is it distinct from them? It would be nice to see this discussion in the main or appendix text.
[2] Yu, Jeremy, et al. "Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems." Computer Methods in Applied Mechanics and Engineering 393 (2022): 114823.
[3] Wang, Sifan, et al. "Learning the solution operator of parametric partial differential equations with physics-informed DeepONets." Science advances 7.40 (2021): eabi8605.
- How were the gradients computed using automatic-differentiation (AD)? I do not see the ML package mentioned, a package like Jax has more accurate and substantially faster AD than one like PyTorch, see [4] Table 6 and Figure 15. The authors may want to consider this to improve performance and reduce the overhead cost for dm and dx label generation.
[4] Jagtap, Ameya D., et al. "How important are activation functions in regression and classification? A survey, performance comparison, and future directions." Journal of Machine Learning for Modeling and Computing 4.1 (2023).
Limitations
- Adequately described in manuscript.