Universal Physics Transformers: A Framework For Efficiently Scaling Neural Operators

Summary

The authors presented Universal Physics Transformers (UPTs), a framework for efficient learning and scaling neural operators. UPTs offer flexibility in handling various data types, whether grid-based or particle-based. UPTs compress data into a low-dimensional latent space and perform dynamics propagation within this space, resulting in fast simulations. Additionally, the latent representation allows evaluation at any point in space-time. UPTs demonstrated strong performance when tested on various fluid dynamics problems.

Strengths

- The authors developed a model based on linear attention that handles both mesh-based and particle-based data effectively. - It leverages a low-dimensional latent space for efficient temporal dynamics propagation, enabling rapid evaluations. - They conducted very interesting Navier-Stokes experiments. - They demonstrated that UPT serves as a robust and efficient baseline. - In scalability experiments, UPT showed excellent performance, scaling efficiently with input size and outperforming other baselines across all tested scales. - They examined the model's convergence concerning the number of I/O points, obtaining good outcomes.

Weaknesses

- The framework is labeled as universal but is only tested on fluid dynamics, lacking experiments on other PDEs. - Although the model scales well with input size, its scalability with model size appears poor based on a conducted experiment (Figure 5). While it outperforms other baselines, its scaling rate is significantly lower than that of models like GINO. Additional experiments are necessary to address this issue (if the UPT is meant to be used in alrge scale applications) - The scalability of UPT concerning training size remains unclear, which is an important property. - The stability of the latent rollout technique for long rollouts is uncertain, raising concerns about its general reliability

Questions

- What do you think causes the performance to plateau with increasing model size as observed in Figure 5? Is it because the error is already sufficiently low, preventing further enhancement? - Why do you believe that latent rollouts do not require stabilization?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors explained well the limitations.

Summary

This paper introduces Universal Physics Transformers (UPTs) to provide a unified learning paradigm for grid- or particle-based structures, enabling scalability across meshes and particles. UPTs mainly follow Encode-Process-Decode paradigm and allow queries at any space-time point through perceiver-like cross attention. To separate the responsibilities of individual components, the authors introduce inverse encoding and decoding losses in addition to the next-step prediction loss. Extensive experiments are conducted in diverse applications, including mesh-based fluid simulations and Lagrangian dynamics.

Strengths

UPTs provide a unified framework that can handle both grid- and particle-based structures, enhancing flexibility across different simulation types. UPTs reduce computational overhead by selecting and aggregating features on supernodes within mesh- and particle-based structures. The paper is well-written with a clear structure.

Weaknesses

- Regardless of the structure of unstructured mesh/structured mesh/point clouds, when processing with UPTs, each node is actually modeled as a token, and a transformer architecture is built on this basis. However, there have been several works [1, 2, 3] that utilize transformers or attention mechanisms to model PDE problems. The novelty of using UPTs is somewhat limited in this way. - The paper misses comparisons to existing work that employs transformer and attention mechanisms for similar purposes. While [1, 2, 3] have designed methods to reduce the computation overhead of transformer or attention mechanisms, the UPTs approach uses a simple random sampling to aggregate supernode features to reduce computation overhead. - While the latent rollout approach significantly accelerates inference speed, the experimental results do not show a clear improvement over the autoregressive unrolling via the physics domain. This raises concerns about the effectiveness and practicality of the latent rollout method. [1] GNOT: A General Neural Operator Transformer for Operator Learning [2] Transolver: A Fast Transformer Solver for PDEs on General Geometries [3] Transformer for Partial Differential Equations' Operator Learning

Questions

- Given that the latent rollout approach does not provide a significant improvement in experimental results compared to autoregressive unrolling, can you elaborate on the potential reasons behind this? Specifically, do you believe that the lack of clear separation between the encoder, approximator, and decoder in this training framework contributes to this issue? Additionally, how effective do you find the inverse encoding and decoding processes? Considering the substantial training overhead and marginal performance gains, what are the key motivations for choosing the latent rollout method over other approaches? Are there any foreseeable optimizations or modifications that could enhance its efficacy? - How dose DiT modulation influences UPTs' performance? Additionally, which features have you found to be most effective as conditions for DiT modulation? - How do you specifically determine the number of supernodes in each region of mesh-structured data? After sampling nodes on the mesh, the original edge connections are lost. What impact does using a radius graph to replace the original mesh connectivity have on the results? Can you provide details on how this transformation affects the model's performance and accuracy? - How is the radius for the radius graph determined when aggregating information at the selected supernodes via a message passing layer? Additionally, do the circles of different supernodes overlap? - For each optimization step, are the supernodes fixed or do they change dynamically? - It is mentioned that the edge limit is imposed on nodes in a radius graph to prevent memory fluctuations. Is the edge limit satisfied if the radius is smaller than the set value, or if the limit is exceeded, are edges randomly removed to comply with the constraint?

Rating

6

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

Yes

Summary

This paper introduces a transformer-based architecture to scale neural operators to larger and more complex conditions involving spatiotemporal modeling. The novelty of this architecture is the use of the so-called latent rollout, which performs autoregressive modeling on the latent space, as opposed to the decoded space in other approaches. Authors claim that this architecture can operate without spatial structures, which is an inherent benefit of transformer-based approaches, and inspection of the latent-space in space-time. The authors show the usefulness of this approach across different physics-based problems. This is a meaningful contribution to design of NNs for physics applications, with potential broader applications in other ML domains. However, there are some questions that this reviewer hopes that the authors can address.

Strengths

1. Paper is clearly written, and appendices help with self-completeness. 2. Proposed method is shown to be better than existing popular architectures for physics-based modeling. 3. Code is provided for reproducibility.

Weaknesses

1. A bit more analysis and information could be included to improve completeness. See Questions. 2. Only test MSE has been used as an accuracy metric. Physics applications typically care about conservation of mass and performance near boundaries. However, the lack of further metrics is somewhat justified by the speedup and memory gains shown,

Questions

1. Authors should be commended on the discussion on memory complexity in Line 105 to 119. However, this could be summarized as a table for cleaner presentation. Can the authors also include time complexity information for a more transparent comparison of the trade-offs associated with the different architectures? (I know this has been somewhat benchmarked in Table 1, but it could be useful to have some more theoretical information) 2. What kind of position encoding is used for the UPT? This is unclear in the paper. Does the type of position encoding matter for Lagrangian vs Eulerian. 3. For a more comprehensive display of results, could the authors include accuracy metrics (e.g., mean error) and memory benefits to Table 1? 4. This has been briefly discussed in Appendix A. But for further clarification, is the present architecture approach limited to physics-related applications? Or are there opportunities for extending the present approach towards other autoregressive ML applications, such as video or language modeling?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

4

Limitations

Limitations are well-discussed in Appendix A.

Summary

The authors present a new framework for unifying PDE surrogate modeling across domains. The proposed encoder, approximator, and decoder structure can accommodate PDEs of different discretizations and simulation types. Solutions are approximated in latent space, which aids in scalability and reducing computational cost of high-dimensional PDE data.

Strengths

- The method for encoding/decoding to a common latent space is exciting and seems to provide a good balance between compute/accuracy. - The experiments are done with challenging fluids problems, which provides a good evaluation of model capabilities. - The authors make a good effort to run a variety of experiments and provide good justification for when certain experiments cannot be performed. Furthermore, it is evident that the authors spent a lot of time thoroughly evaluating different model choices/hyperparameters and their effects. - The generalization study and latent space evaluation are convincing. - The authors consider Lagrangian fluid simulation, which is not commonly done in PDE surrogate modeling, and being able to accommodate this underexplored domain is good. - The authors provide a good background of the fluid data used and the clarity of the paper is good.

Weaknesses

- The Unet, FNO, and GINO benchmarks are good, however, they are not specifically tailored for efficient computation. There have been prior works that address latent space modeling or efficient PDE modeling of 3D data, namely Latent Spectral Models [1], Latent Evolution of PDEs (LE-PDE) [2], and FactFormer [3]. I would like to see some reasoning or experiments as to why your model is better. - The proposed architecture seems similar to OFormer [4]. It follows a similar encoding/propagating/decoding scheme as well as uses the transformer architecture. There are good improvements to adjust to higher dimensional and diverse data (e.g, Perceiver, supernodes, decoder design), but the broad architectural claims are shared with this previously proposed model (e.g. using a latent space, training an encoder/decoder, relying on transformer models for scalability). There are also other transformer-based works that present scalable architectures (DPOT [5], MPP [6]). I understand that these models do not include mechanisms for high-dimensional or irregular data, but I am curious to see how your model compares to these transformer-based models on a regular domain. - There is no justification given for why your model is a neural operator. I know that not every paper that proposes a neural operator justifies the operator approximation theorem, and there are good discretization invariance results, but I would like to see some reasoning for how your model satisfies operator approximation. Overall, I think the paper is a good contribution and is a valuable step in extending neural operators to large, complex PDE problems. There is a lack of comparable benchmarks, however this does not significantly detract from the experimental rigor and results. 1. Haixu Wu, Tengge Hu, Huakun Luo, Jianmin Wang, Mingsheng Long, Solving High-Dimensional PDEs with Latent Spectral Models. https://arxiv.org/abs/2301.12664 2. Tailin Wu, Takashi Maruyama, Jure Leskovec, Learning to Accelerate Partial Differential Equations via Latent Global Evolution, https://arxiv.org/abs/2206.07681 3. Zijie Li, Dule Shu, Amir Barati Farimani, Scalable Transformer for PDE Surrogate Modeling, https://arxiv.org/abs/2305.17560 4. Zijie Li, Kazem Meidani, Amir Barati Farimani, Transformer for Partial Differential Equations' Operator Learning, https://arxiv.org/abs/2205.13671 5. Zhongkai Hao, Chang Su, Songming Liu, Julius Berner, Chengyang Ying, Hang Su, Anima Anandkumar, Jian Song, Jun Zhu, DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training, https://arxiv.org/abs/2403.03542 6. Michael McCabe, Bruno Régaldo-Saint Blancard, Liam Holden Parker, Ruben Ohana, Miles Cranmer, Alberto Bietti, Michael Eickenberg, Siavash Golkar, Geraud Krawezik, Francois Lanusse, Mariel Pettee, Tiberiu Tesileanu, Kyunghyun Cho, Shirley Ho, Multiple Physics Pretraining for Physical Surrogate Models, https://arxiv.org/abs/2310.02994

Questions

- I’m curious about the general behavior of stability of error accumulation of your model during auto-regressive rollout. Are there specific training details that you use to mitigate this? Does using a latent space help mitigate this effect? - What are your thoughts on using a latent space vs. directly working in physics space when it comes to the accuracy of the solutions? Latent unrolling is obviously faster during inference, but even in the physical domain many neural surrogates are still orders of magnitude faster than numerical solvers with the added benefit of being simpler and potentially more accurate than working in a latent space.

Rating

7

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

- For common benchmark problems on a regular grid, I don’t see why this architecture would be preferred over a baseline or fine-tuning a pretrained model. Most physical problems use a mesh or irregular grid, but certain problems such as weather or isotropic turbulence come to mind. - Fixing the latent space is presented as a method to standardize different physics to a common space. However, more complex physics problems (3D turbulence vs. 2D laminar flows) may conceivably require larger latent spaces and the model would need to be retrained for different latent sizes.

Summary

In this paper a framework for efficiently scaling neural operators is introduced under the name of Universal Physics Transformers (UPTs). It is a novel paradigm that scales neural operators across diverse spatio-temporal problems without relying on specific grid or particle-based structures. Leveraging transformer architectures, UPTs propagate dynamics within a compressed latent space, enabling efficient and flexible simulations.

Strengths

1/ Originality: The paper presents an original contribution to the field of neural operators by introducing Universal Physics Transformers (UPTs). This novel paradigm removes the traditional reliance on grid or particle-based latent structures, enabling greater flexibility and scalability across various simulation types. The innovative use of transformer architectures to propagate dynamics within a compressed latent space is a quite original combination of existing ideas applied to a new domain. 2/ Quality: The quality of the overall contribution is good. The authors address a critical challenge in scaling neural operators for complex simulations and provide a robust framework with practical applications. The UPT framework is well-formulated, and the encoding and decoding schemes are designed to ensure efficient simulation rollouts. The experiments are comprehensive, covering multiple types of simulations, and the results demonstrate the superiority of UPTs in terms of performance and scalability. Moreover, the paper's methodology is sound, and the technical claims are well-supported by evidence from the experiments. 3/ Clarity: The paper is well-written and clearly presented. The authors provide a thorough background. 4/ Significance: The significance of the paper lies in its potential impact on the broader scientific and engineering communities. By providing a unified and scalable framework for neural operators, UPTs can be applied to a wide range of spatio-temporal problems, including those in fluid dynamics. The ability to efficiently handle large-scale simulations can lead to significant advancements in these fields, offering valuable insights and solutions to complex physical phenomena. Moreover, the experimental design is robust, with appropriate datasets and baselines used for comparison. The methodology is clearly described, allowing for reproducibility of the results. The use of inverse encoding and decoding techniques to enable latent rollouts is particularly noteworthy, as it demonstrates a deep understanding of the underlying principles and challenges in neural operator learning. 5/ Latente space and scalability: The Universal Physics Transformers (UPTs) framework is efficient since it allows to get compact yet expressive latent space representation, capturing the essential dynamics of physical systems while significantly reducing memory usage and computational overhead. This compactness, combined with efficient encoding and decoding schemes, ensures that the transformation to and from the latent space is both efficient and accurate. By unifying the encoding of various grids and particles, UPTs simplify the modeling process and improve generalization across different spatio-temporal problems. The empirical validation provided in the paper, through steady-state and transient flow simulations, demonstrates that UPTs achieve lower mean-squared error (MSE) and faster computation times compared to other models, highlighting the effectiveness of the latent space representation. In terms of scalability, UPTs are designed to handle large-scale simulations effectively. The fixed-size latent space, regardless of input size, allows UPTs to manage large inputs without a proportional increase in computational cost. This scalability is evident in the framework's ability to maintain performance and efficiency with large meshes and numerous particles. The efficient latent space rollouts further enhance scalability by enabling quick updates and predictions, making UPTs suitable for time-dependent simulations. Additionally, the transformer architecture provides a robust foundation for scalability, ensuring that UPTs can handle complex spatio-temporal simulations efficiently. 6/ Lagrangian dynamic modeling: The Universal Physics Transformers (UPTs) framework presents several notable strengths when viewed from the perspective of Lagrangian dynamic modeling. One of the primary advantages is the framework's ability to model particle-based simulations effectively without relying on traditional particle-structures. In Lagrangian simulations, particles move with the local deformation of the continuum, and modeling these dynamics accurately requires handling a large number of particles and their interactions. UPTs use a unified latent space to encode particle information, enabling the framework to handle varying numbers of particles flexibly. This flexibility is particularly beneficial for Lagrangian methods, such as Smoothed Particle Hydrodynamics (SPH), where particle counts can vary significantly based on the simulation's complexity. By compressing this information into a fixed-size latent space, UPTs manage to reduce computational overhead while maintaining the ability to capture intricate particle interactions. The efficient encoding and decoding processes in UPTs ensure that particle dynamics are propagated accurately and swiftly within the latent space. This efficiency is crucial for large-scale Lagrangian simulations, where the computational cost can be prohibitive with traditional methods. The ability to perform latent rollouts enables UPTs to predict future states quickly, making the framework suitable for real-time or near-real-time applications in Lagrangian dynamic modeling. The paper provides strong empirical evidence of UPTs' effectiveness in Lagrangian dynamic modeling through experiments on datasets such as the Taylor-Green vortex in three dimensions (TGV3D). The results demonstrate that UPTs can effectively learn the underlying field dynamics and predict particle velocities with lower error compared to traditional Graph Neural Networks (GNNs) and other baselines. This empirical validation underscores the practical applicability of UPTs in complex Lagrangian simulations UPTs exhibit strong generalization capabilities, which are critical for Lagrangian dynamic modeling. The ability to query the latent representation at any point in space-time allows UPTs to adapt to different particle distributions and simulation conditions without extensive retraining. This robustness ensures that UPTs can handle a wide range of Lagrangian problems, from small-scale particle systems to large-scale simulations involving a representative number of particles. The use of transformers to manage the latent space representation in UPTs is a significant technical innovation. Transformers are known for their efficiency in handling large datasets and sequences, and their application in UPTs leverages this strength to manage the complexities of Lagrangian dynamics. This choice of architecture allows UPTs to model particle interactions accurately while maintaining computational efficiency.

Weaknesses

1/ Clarity and detail in methodology: The methodology, particularly the detailed implementation of the encoding and decoding schemes, could be more clearly articulated. Some parts of the algorithms are difficult to follow. Providing additional diagrams, detailed explanations, or pseudo-code would help clarify these complex processes. Ensuring that each step of the process is well-explained and easily understandable would make the paper more accessible and replicable. 2/ Scalability with extremely large datasets: While the paper demonstrates UPTs' scalability, there is limited discussion on how the framework performs with extremely large datasets or in distributed computing environments. Given the increasing size of datasets in fields like high-resolution climate modeling and genomics, a discussion or preliminary results on the scalability of UPTs in such contexts would be valuable. This could include potential challenges, proposed solutions, and the impact on computational resources. 3/ Insufficient analysis of generalization capabilities: While UPTs are shown to generalize across different simulation types, the paper lacks a deep analysis of this capability. Neural operator networks are often praised for their ability to generalize across different boundary conditions and initial states. A more thorough examination of UPTs' generalization performance, especially in unseen or out-of-distribution scenarios, would strengthen the claims. Detailed experiments and discussions on how UPTs perform under varied conditions would be valuable. 4/ Potential overfitting concerns: Given the complexity and high capacity of transformer-based models, there is a risk of overfitting, especially on smaller datasets. The paper briefly mentions overfitting issues in some experiments but does not provide a detailed strategy for mitigating this. More discussion on regularization techniques, data augmentation strategies, or how UPTs handle overfitting would be beneficial. Insights into how the model can be generalized better and made more robust against overfitting are crucial for practical applications. 5/ Analysis from particle and grid based methods: The Universal Physics Transformers (UPTs) framework offers a novel approach, but there are specific areas where it could be improved, particularly when compared to traditional particle and grid-based methods. A/ Lack of detailed comparison with established methods: The paper does not provide a thorough comparison with well-established particle-based methods (such as Smoothed Particle Hydrodynamics, SPH) and grid-based methods (such as Finite Volume Methods, FVM). Including detailed performance metrics, such as accuracy, computational cost, and scalability, would help highlight UPTs' advantages and areas needing improvement. Specifically, comparative studies on benchmark problems typically addressed by SPH (except figure 12) and FVM would provide more context. B/ Handling of boundary conditions: Particle and grid-based methods have well-established techniques for handling complex boundary conditions, which are often crucial in physical simulations. The paper does not sufficiently address how UPTs manage complex boundary conditions compared to these traditional methods. A more detailed discussion or experiments showcasing UPTs' effectiveness in dealing with various boundary conditions would strengthen the paper's claims. C/ Adaptation to high-resolution grids and large particle systems: While UPTs are shown to be scalable, the paper lacks detailed insights into their performance on extremely high-resolution grids or very large particle systems. Traditional methods often excel in these areas due to their specialized structures and optimizations. Providing more evidence on how UPTs handle such scenarios, including any potential bottlenecks and solutions, would be beneficial. D/ Numerical stability and accuracy: Grid-based methods, such as FVM, are known for their numerical stability and accuracy, especially in simulating fluid dynamics. The paper does not provide an in-depth analysis of UPTs' numerical stability and accuracy compared to these methods. Detailed experiments and discussions on how UPTs ensure stability and accuracy over long simulation times would enhance the paper’s credibility. E/ Computational efficiency in complex geometries: Particle and grid-based methods have specific strategies to efficiently handle complex geometries, such as adaptive meshing in grid-based methods or kernel adjustments in particle methods. The paper does not clearly explain how UPTs manage complex geometries and whether they can maintain computational efficiency in such scenarios. More detailed experiments or case studies involving complex geometrical domains would be informative. F/ Interoperability with existing simulation tools: Traditional methods are often integrated into comprehensive simulation tools (e.g., OpenFOAM for grid-based methods or LAMMPS for particle-based methods). The paper does not discuss how UPTs can be integrated or used alongside these existing tools, which is crucial for practical adoption. Providing insights into interoperability and how UPTs can complement or enhance traditional methods within established simulation frameworks would be useful. G/ Handling of multiscale phenomena: Particle and grid-based methods have developed sophisticated techniques to handle multiscale phenomena, such as adaptive mesh refinement (AMR) in grid-based methods. The paper lacks a discussion on how UPTs address multiscale phenomena, which are common in many physical simulations. Including experiments or theoretical discussions on UPTs' capabilities in multiscale modeling would be beneficial. 6/ Model conditioning standpoint: Model conditioning is crucial for ensuring that neural networks adapt accurately to varying inputs and simulation conditions. The Universal Physics Transformers (UPTs) framework has room for improvement in this aspect. Here are specific areas where the paper could enhance its discussion and implementation of model conditioning. A/ Insufficient detail on conditioning mechanisms: The paper briefly mentions the use of feature modulation (e.g., DiT modulation) for conditioning UPTs to various inputs, such as the current timestep and boundary conditions. However, it lacks a detailed explanation of how these conditioning mechanisms are implemented and their impact on model performance. Providing more comprehensive descriptions and theoretical justifications for the chosen conditioning methods would help readers understand their effectiveness and potential limitations. B/ Limited analysis of conditioning performance: While the paper demonstrates that UPTs can handle different flow regimes and domains, it does not provide a thorough analysis of how well the conditioning mechanisms work across a broader range of scenarios. Including detailed experiments that specifically evaluate the performance of UPTs under different conditioning inputs, such as varying boundary conditions, initial states, and external forces, would strengthen the paper's claims. C/ Scalability of conditioning methods: The scalability of the conditioning mechanisms is not thoroughly discussed. As simulations grow in complexity, the effectiveness of conditioning methods can degrade if not properly scaled. The paper should address how the conditioning mechanisms scale with increasing input dimensions and simulation complexities, providing insights into any potential bottlenecks and how they are mitigated. D/ Generalization to unseen conditions: One of the strengths of neural operator networks is their ability to generalize to unseen conditions. The paper does not sufficiently explore how well UPTs generalize to entirely new boundary conditions or physical scenarios that were not present in the training data. Including experiments that test the generalization capabilities of UPTs to unseen conditions would provide valuable insights into the robustness of the conditioning methods. E/ Comparison with other conditioning approaches: The paper does not compare its conditioning mechanisms with other advanced conditioning techniques used in neural operator networks or related fields. A comparative analysis of different conditioning approaches, such as those used in Fourier Neural Operators and DeepONets, would highlight the strengths and weaknesses of the methods employed in UPTs. F/ Impact of conditioning on training stability: Conditioning mechanisms can significantly impact the training stability of neural networks. The paper does not discuss how the chosen conditioning methods affect the stability of UPT training, especially in the presence of complex and noisy data. Analyzing and addressing potential stability issues arising from conditioning would be beneficial. G/ Real-world application scenarios: While the paper presents conditioning in the context of fluid dynamics simulations, it does not discuss its applicability to other real-world scenarios that require complex conditioning. Exploring and providing evidence of UPTs' effectiveness in diverse applications, such as climate modeling or structural analysis, where conditioning to various external factors is crucial, would enhance the practical relevance of the framework.

Questions

1/ The paper briefly mentions feature modulation for model conditioning but lacks detailed discussion on handling complex boundary conditions. Providing specific examples or additional experiments that illustrate UPTs' effectiveness in managing complex boundary conditions would strengthen the paper’s claims. Addressing this can clarify the practical applicability of UPTs in real-world scenarios where boundary conditions play a crucial role. How does the UPT framework handle complex boundary conditions compared to traditional particle and grid-based methods? 2/ While UPTs are presented as efficient and scalable, the paper does not delve into the specifics of their numerical stability and accuracy compared to traditional methods. Discussing strategies to maintain stability and accuracy over extended simulations and providing relevant experimental evidence would address potential concerns about the reliability of UPTs in long-term applications. What measures are in place to ensure the numerical stability and accuracy of UPTs over long simulation periods? 3/ The paper demonstrates scalability and generalization within certain limits but does not extensively explore these aspects in more demanding scenarios. Providing additional experimental results or theoretical insights on UPTs’ performance in high-resolution and large-scale environments, as well as their ability to generalize to entirely new conditions, would significantly bolster the paper’s contribution and practical relevance. What are the scalability and generalization capabilities of UPTs when applied to extremely high-resolution grids, large particle systems, and unseen physical scenarios?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Reply to rebuttal

Thank you for the clarifications, and the additional experiments in general response which I believe strengthen the paper. I have raised my score. However, it would be better if the supernode pooling/radius graph construction could be included either in the maintext or supplementary for reference.

Summary of the revision and follow-up

Thank you for addressing the feedback provided in my initial review. Here is a summary of the revisions and responses you have made, along with some additional feedback and observations. **Scalability to extremely large datasets / high-resolution grids** I commend your efforts to demonstrate scalability within the constraints of your computational resources. The additional experiments on the Shallow Water and Navier-Stokes datasets effectively illustrate your framework's ability to handle large datasets. Your discussion on using distributed training across multiple GPUs highlights the potential of your approach in handling complex simulations. **Generalization capabilities** The expanded tests on generalization, including across different flow regimes and discretizations, strengthen the evidence for your framework's robustness. This comprehensive analysis is a significant enhancement and effectively addresses previous concerns about the adaptability of your model. **Handling of boundary conditions** The examples provided on handling complex boundary conditions through experiments like the ShapeNet car and transient flow simulations highlight the flexibility and adaptability of UPTs in various scenarios. This aspect of your work is well-explained and a notable strength of your framework.

Suggestions

1. While real-world applications are beyond the current scope, discussing potential use cases or hypothetical scenarios could highlight the practical impact of UPTs. 2. Consider exploring how UPTs might be integrated with traditional particle or grid-based methods in practice. Discussing potential hybrid approaches could broaden the applicability of your work 3. Discussing specific measures or techniques to enhance long-term stability and robustness in simulations, particularly with noisy data, could strengthen the paper's claims. 4. As hardware capabilities evolve, exploring how UPTs can leverage advancements in GPU and TPU technologies could provide insights into the future potential of your framework.

Follow-up questions

1. In your response, you mentioned enhancing the model conditioning with techniques such as feature modulation, but I could not find the additional details in the paper. Could you elaborate on how the DiT modulation and any other conditioning mechanisms are implemented in the UPT framework, and how these techniques specifically improve model adaptability to varying boundary conditions and simulation inputs? 2. You highlighted the use of transformers to manage complex boundary conditions, particularly in experiments like the ShapeNet car. Could you provide more details on how the transformers are configured to encode these boundary conditions effectively? Are there specific architectural adjustments or tokenization strategies employed to ensure accurate representation and processing of boundary conditions? 3. In the context of your framework's numerical stability, you mentioned that UPT rollouts are stable without specific measures. Could you clarify how the UPT framework maintains stability and accuracy over extended simulations, especially when compared to traditional numerical solvers that require small timesteps for stability? Are there specific aspects of the transformer architecture that contribute to this stability? 4. While you provided insights into UPT's scalability with large datasets and distributed training, can you discuss any potential limitations or bottlenecks encountered when scaling the model further? How does the framework handle challenges such as increased computational demands or memory usage as the complexity of the simulations and datasets grows?

Missing pseudo-code and extra details on model conditioning

I am unable to find the pseudo-code and the additional details on model conditioning that are mentioned in your response.