Inferring Hybrid Neural Fluid Fields from Videos

Summary

The paper proposes a novel approach for recovering the density and velocity fields of inviscid fluids from sparse multiview videos. The model has two main contributions: - First, it incorporates physics-based losses to enforce the inference of a physically plausible velocity field that is divergence-free and drives the transport of density. This helps to deal with the visual ambiguities of fluid velocity. - Second, the model provides a hybrid neural velocity representation, which consists of a base neural velocity field capturing most irrotational energy and a vortex particle-based velocity modeling residual turbulent velocity. This representation enables the recovery of vortical flow details.

Strengths

Originality: This paper primarily tackles the visual ambiguities in inverse rendering techniques for fluid data, focusing on resolving visual ambiguities. The integration of physics-based losses into the volume rendering framework is a rational approach. The results in Figure 8 illustrate the effectiveness of the newly proposed learning constraints in significantly improving the accuracy of the reconstruction. Moreover, the introduction of a hybrid representation of velocity fields and particle-based vortex flow showcases originality in the methodology. Significance: The proposed model makes a significant contribution to the visual understanding of fluids, particularly smoke, fog, and gas.

Weaknesses

For methodology: 1. One contribution of this paper is to incorporate new forms of physical constraints in the framework of inverse rendering. However, the general idea is not entirely novel as previous work by Chu et al. (2022) has also presented a similar (albeit different) approach, which somewhat weakens the technical novelty of this paper. For example, using the density transport equation from incompressible fluid or NS equation as a loss function has been employed in other related papers. Please refer to the work from Baieri et al. (2023) and Li et al. (2023). - [Baieri et al., 2023] Fluid Dynamics Network: Topology-Agnostic 4D Reconstruction via Fluid Dynamics Priors. Arxiv, 2023. - [Li et al., 2023] PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification. ICLR, 2023. 2. The constraint imposed on emitting radiance in HyFluid does present limitations in its applicability to real-world scenarios. The requirement of a constant emitting radiance may hinder accurate recovery in situations where spatially-varying lighting exists in the scene. Lacking references: 3. Some closely related work seems to be missing, such as NeuroFluid (Guan et al., 2022) and PAC-NeRF (Li et al., 2023), both of which also focus on visual physical inference through inverse rendering. It is important to acknowledge these works as they contribute to the existing body of literature in this field and provide valuable insights and techniques for comparison and benchmarking purposes. - [Guan et al., 2022] NeuroFluid: Fluid Dynamics Grounding with Particle-Driven Neural Radiance Fields. ICML, 2022. For experiments: 4. Is the proposed method limited to handling only inviscid fluids? It would be beneficial to evaluate the proposed method in a broader range of fluid scenarios, including different types of flows (e.g., laminar, turbulent) and varying fluid properties (e.g., viscosity, density). This would demonstrate the generalization ability of HyFluid under diverse conditions. 5. Considering that the constraint of constant radiance may not be practical in complex real-world scenes, I highly recommend that the authors compare the reconstruction results obtained with and without the constraint. 6. The proposed model is only compared with two existing models, which is not sufficient. To make the results more convincing, the authors could incorporate more advanced neural rendering techniques designed specifically for dynamic scenes. Additionally, given that the dataset comprises synthetic fluid simulation data, it would be beneficial for the authors to provide quantitative results regarding the reconstruction of the velocity field in comparison to the ground truth on these simulated scenes.

Questions

1. In the paper, a grid-based representation is used for density and velocity. When implementing $L_{density}$ and $L_{project}$, how exactly are these calculations performed? Are the loss functions computed for all grid positions, or is interpolation used to compute the losses at sampled points in space? 2. What is the difference in performance between decomposing the velocity field into a base neural velocity field and a vortex particle-based velocity, versus solely using high-frequency position embedding? 3. In the case of using physics-based loss functions in HyFluid, how does the predicted velocity field of HyFluid compare to the ground truth (GT) velocity field quantitatively? Can HyFluid be compared to other methods in terms of velocity field quantitatively? 4. The paper does not provide experimental results and discussions regarding the constraint of constant radiance. What is the capability of HyFluid with real-world scenes that exhibit spatially-varying radiance? 5. Can you provide more quantitative/ qualitative results of other baselines?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Summary

The paper presents a neural reconstruction method for individual fluid flows which is re-trained for each new scene. It combines the established iNPGs for representing densities and velocities with vorticity transporting partices in a hybrid flow representation called HyFluid. A set of appearance- and physics-based losses is used to train the iNPGs and optimize vorticity and lighting in 3 steps. With this approach it is possible to recover 3D density and velocity from sparse views and enable novel-view rendering, re-simulation, and predictions of states over time.

Strengths

I see the following strong points in this submission: - The method is build on physical priors to recover a physically meaningful velocity field. - The loss to the projected velocity to ensure divergence freeness is novel as far as I can tell. - Using particles to transport voriticity seems novel, at least in that context. - Provides better results then SOTA neural methods regarding re-simulation and prediction. - The results look good, the motion of the density is coherent and also suited for re-simulation. - The reconstruction works with simple, sparse images.

Weaknesses

The treatment for vorticity seems novel, but from the provided ablations I cannot see a substantial impact: - In Fig. 8 there seems to be a slight improvement. - In the videos, the ablation study results of "w/o vort." and "full" look very similar. - The difference could be more visible in the flow field, and but this is not demonstrated. - In the videos, the novel view synthesis results of PINF [1] and HyFlow are almost identical.

Questions

- Since the approach is trained on ScalarFlow [2] it would be interesting to see how the method compares to the ScalarFlow reconstructions. ScalarFlow is not using neural networks, and as such a good not-learned baseline as a single scene reconstruction method using physical priors (advection, pressure projection). Re-simulation and future prediction might also be possible there since ScalarFlow is using a fluid solver (MantaFlow) internally. Can the authors explain whether they haven't compared to the ScalarFlow 3D data despite using the 2D input images? - It would also be interesting to compare to GlobTrans [3] as this method it is trained/optimized as a re-simulation method. Have the authors looked into this? - The paper uses use an unseen GT view to evaluate the metrics. Have the authors tried/considered reconstructing a synthetic case to be able to make a 3D comparison? This might also show the efficacy of the flow reconstruction more clearly, since the visual appearance could be matched perfectly. - The velocity flickers in the supplemental video, there is no smooth evolution. Is there any coupling of the velocity over time (like self-advection)? - In the videos the velocity fields are rendered to 2D. Can the authors explain how is that done? Wouldn't a slice provide more insight in the actual motions? Averaging might blur the small-scale vorticities. Overall, I think the paper targets an interesting direction and provides an interesting approach for a tough problem. I still somewhat on the edge regarding this paper, but I'm open to readjusting/raising my score after the rebuttal. References: [1] Mengyu Chu, Lingjie Liu, Quan Zheng, Erik Franz, Hans-Peter Seidel, Christian Theobalt, and Rhaleb Zayer. 318 Physics informed neural fields for smoke reconstruction with sparse data. ACM Transactions on Graphics, 2022. [2] Marie-Lena Eckert, Kiwon Um, and Nils Thuerey. Scalarflow: a large-scale volumetric data set of real-world 330 scalar transport flows for computer animation and machine learning. ACM Transactions on Graphics (TOG), 2019. [3] Erik Franz, Barbara Solenthaler, and Nils Thuerey. Global transport for fluid reconstruction with learned 337 self-supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Limitations are briefly discussed, I don't see anything significant missing here.

Summary

This paper presents an innovative neural dynamic reconstruction method that achieves good results in recovering fluid density and velocity fields through the introduction of physical constraints and a hybrid neural velocity representation. Despite some weaknesses and issues, this method has significant implications for the in-depth study and resolution of fluid dynamics problems. Future work could focus on refining the method and validating and applying it to a wider range of fluid scenarios.

Strengths

This method proposes a new approach to neural dynamic reconstruction that can simultaneously recover fluid density and velocity fields, overcoming the challenges posed by the visual ambiguities of fluid velocity in existing methods. Physics-based losses are introduced to enforce a divergence-free velocity field, driving the transport of density and enhancing the accuracy of velocity estimation. A hybrid neural velocity representation is designed, incorporating a base neural velocity field that captures most irrotational energy and a vortex particle-based velocity that models residual turbulent velocity. This enables the method to effectively recover vortical flow details.

Weaknesses

The paper lacks detailed presentation of experimental results and quantitative evaluations, thus lacking sufficient validation of the method's performance. There is no comparison with other methods in addressing the visual ambiguity of fluid velocity, making it difficult to assess the method's advantages and disadvantages comprehensively.

Questions

Further investigation is needed to determine whether the method can handle complex fluid scenarios such as non-Newtonian or multiphase fluids in practical applications.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The paper does not mention the computational resource requirements, such as computation time and memory consumption, which are important factors for the feasibility of practical applications.

Summary

This paper works on reconstructing fluid density and velocity from multi-view videos. The main idea is to inject visual clues with NeRF. They propose some physics-based regularization terms to deal with visual ambiguity that video can not reflect the inner fluid states.

Strengths

Good motivation: Visual ambiguity indeed is the critical problem that needs to be solved in this setting. Novel framework: Introducing physical losses is a not-easy but natural point. The introduced losses are general to inspire other related researchers.

Weaknesses

1. Setting is not new. Indeed, reconstructing fluid velocity and density from videos has been studied in NeuroFluid (ICML 2022). I didn't find the introduction or comparison. Why? 2. The proposed method is not convincing to me. - The proposed physics-based losses can only play a regularization role in the training. To solve the visual ambiguity, the framework must introduce direct supervision of the fluid velocity. Nevertheless, ambiguity still exists. - And, the key question is how to get the initial state of the fluid, which decides the performance of this work. The authors **should** clarity this point. 3. Experiments are incomplete, lacking key results. - Experiments just show the render results and velocity results. How about density? The authors claim that they can recover fluid density but not shown in the experiments. - How did you evaluate the render results? Directly render fluid from the view angle used in the training stage? Please claim them before your experiments. - Only evaluate the stoke with a similar environment. Can you show more examples, with more complex shapes and materials?

Questions

For details, please refer to the weaknesses section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

yes

Summary

In this paper, the authors propose a method (HyFluid) to infer the fluid density and velocities from multiview videos. To deal with the visual ambiguities of fluid, physics-losses are introduced to try to enforce physics plausible velocities. A neural velocity field and a vortex particle-based velocity are introduced to capture the irrotational energy and model residual turbulent velocity respectively. Th experiments show that the proposed method has the ability to recover the vortical flow details to some extent.

Strengths

The paper is overall well written and organized. The proposed density loss and projection loss are well derived from the fluid mechanics. The experiments show that the proposed method can perform better than the baselines.

Weaknesses

The authors claim that the method can give physics plausible estimation of fluid fields. The physics intuitions for the density loss and projection are clear according to the incompressible condition. However, I don't see obvious physics intuition for the laminar loss. For the problem that visual appearance of fluids depends on lighting and fluid substance properties, there is no further exploration in this direction. The problem becomes more important when the fluid is liquid.

Questions

- Does the MGPCG applied to every simulation step to project the velocity during the training? How is the runtime cost for computing the projected velocity? - Why to encourage high-density regions to have non-zero velocity (line 137)? What is the physics intuition here? - As the visual appearance of fluid highly depends on the rendering process, does the results very sensitive to the parameters of the renderer?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

yes

Thanks for your response!

Dear author, Thanks for your efforts, which answer most of my questions. But, there are two questions I need to discuss further. 1. I think only testing on the ScalarFlow dataset is limited. Could you compare with PINF on other real scenes, such as predicting water? Otherwise, you can explain why evaluating on the ScalarFlow is sufficient. Minor: could you please show some image samples of the scene you used? 2. Could you please describe the angle interval of the 5 views in the novel-view synthesis? Thanks!

Post rebuttal

I thank the authors for the clarifications, good to see the synthetic results. Unlike mentioned in other reviews, this now shows both real and synthetic results, so I’d be happy to support acceptance, and I’ll raise my score.

Thanks for the reply

Thank you for the replies to my questions and comments. After reading the other reviews and answers, most of my concerns are addressed. I'll raise my score on this paper and look forward to its future updates with improved content.