DiffSF: Diffusion Models for Scene Flow Estimation

Summary

This paper introduces a novel diffusion model designed for scene flow estimation, aiming to enhance both accuracy and robustness, particularly in the presence of noisy inputs or occlusions. The proposed denoising diffusion models effectively handle noisy data by modeling stochastic processes, filtering out sensor-induced noise, and focusing on underlying patterns. This approach not only improves prediction accuracy but also introduces a measure of uncertainty by averaging over multiple hypotheses without altering the training process. Additionally, the paper presents a new architecture that integrates transformers with diffusion models, further boosting accuracy and robustness across various benchmark datasets.

Strengths

1. The motivation for using diffusion models in scene flow estimation is well-justified. Real-world data often presents challenges such as severe occlusion and noisy inputs, necessitating robust models for scene flow estimation. 2. The proposed method demonstrates state-of-the-art performance through extensive experiments on multiple benchmarks, including FlyingThings3D, KITTI Scene Flow, and Waymo-Open. 3. The paper is well-written and easy to understand, making the concepts accessible to readers. 4. The reproducibility of the results is strong, ensuring that the findings can be reliably replicated.

Weaknesses

The paper lacks qualitative results. I would've liked to see visual comparison of scene flow results (especially for challenging cases) on various datasets. This makes it hard to interpret where the proposed approach performs better than current methods and more importantly where the proposed approach fails.

Questions

1. Line 250 states "showcasing DiffSF’s effectiveness on real-world data.." Can the authors provide some challenging samples of data that your model is robust against? 2. Which datasets are used in generation of results for figures 3 and 4?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Authors have presented the limitations of the work (which is a good thing), but this has been mostly discussed in the supplementary section (which is outside of the main paper). I would encourage the authors to include a brief summary of the limitations in the main paper as well.

Summary

This paper deals with scene flow estimation in 3D point clouds. It proposes a formulation based on diffusion model. The model takes the source and target frames of 3D point clouds as condition and turn the problem into a conditional generation problem. Different from naive conditional generation, the formulation considered the characteristic of the scene flow problem to make each reverse diffusion step as a "flow adjustment". Experiments are done on standard benchmarks.

Strengths

- The proposed formulation considered the characteristic of the scene flow problem to make each reverse diffusion step as a "flow adjustment". It makes so much sense and is a elegant solution to this problem. - The presentation of this paper is clear and easy to follow. - The experiment results significantly outperform existing baselines. - Code and visualization are provided.

Weaknesses

- The author could consider providing more visualizations of the (reverse) diffusion process other than the only one GIF attached in the supplementary. This will be very helpful.formulation - Minor: the citation notation seems to be wrong. For example, citing paper 1 should be "[1]", not "(1)". - Minor: in Tables 1, 2 and 6, there are numbers in both bold and underlines. What are underlined numbers mean? This should be explained.

Questions

What is the definition of "uncertainty"? I understand "EPE" and "outlier". Is there a definition of "uncertainty" somewhere in the paper?

Rating

8

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

Limitation was mentioned. No other limitation is found besides the ones mentioned in the paper.

Summary

This paper introduces DiffSF, which integrates transformer-based scene flow estimation with denoising diffusion models. The diffusion process involves progressively perturbing the ground truth scene flow vector field by adding Gaussian noise. Conversely, during the reverse process, the scene flow vector field prediction is recovered from randomly sampled Gaussian noise by conditioning on a source and a target point cloud. The DiffSF achieves state-of-the-art performance on FlyingThings3D, KITTI Scene Flow, and Waymo-Open.

Strengths

Extending denoising diffusion models for scene flow estimation is a novel approach that is straightforward to understand. The ablation study is comprehensive, such as evaluating the impact of varying the number of time steps during both training and sampling for the diffusion model.

Weaknesses

The tech contribution is limited. This paper does not invent a new concept given a relevant work of optical flow estimation using diffusion models [1]. To enhance understanding of the diffusion process for scene flow estimation, visualizations of source points with added noise at different time steps should be included. The study is limited to the GMSF model architecture. It is strongly recommended to evaluate other popular architectures, as the key concept of the paper is to introduce the diffusion model for scene flow estimation. Additionally, it would be valuable to investigate whether this approach works in the context of self-supervised scene flow estimation such as [2][3]. [1] The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation [2] Self-Supervised Robust Scene Flow Estimation via the Alignment of Probability Density Functions [3] Just Go with the Flow: Self-Supervised Scene Flow Estimation

Questions

It would be beneficial to include a discussion or exploration on the effects of adding noise directly to the ground truth scene flow vector field versus adding noise to the embedding of the ground truth scene flow vector field, as achieved by a separate network such as a VAE model.

Rating

7

Confidence

5

Soundness

2

Presentation

2

Contribution

2

Limitations

N/A

Summary

This paper proposes a scene flow estimation method that estimates scene flow from point clouds. The proposed method combines a previous scene flow method, GMSF, and diffusion model, where a denoising block is introduced based on GMSF. During training, the source point cloud is first warped via noisy scene flow, and then the warped source point cloud and the target point cloud are fed to the denoising block. Experimental results demonstrate the proposed method achieves the best performance compared with state-of-the-art approaches. In addition, the paper shows interesting results that the proposed method can be used to measure the hardness of input point clouds. That is, given a pair of point clouds, the proposed method can obtain multiple scene flow results using different initial randomly sampled noise, where the uncertainty (standard deviation) of scene flow results is highly correlated with their accuracy.

Strengths

1. The idea of the proposed method is interesting and sounds good. 2. The result of uncertainty-error correspondence is interesting, indicating the proposed method can be used to predict the hardness (estimation difficulty ) of the input point cloud pair. 3. Extensive experimental results shows the effectiveness of the proposed method

Weaknesses

1. The organization of the technical part can be improved. It is a bit difficult to follow Sec. 3.2. For example, the paper has already introduced the forward process, reverse process and the training loss, and then introduces the details of the reverse process again. 2. The proposed method uses the same feature extractors with shared weights to extract features from the warped and target point clouds. However, the distributions of the warped point cloud may be rather different from the target one, since the warped point cloud is noisy and the target point cloud is clean. Would such differences negatively affect the training of feature extractors? Similarly, the training of the linear layers in the Local-Global-Cross Transformer may also be negatively affected. If the feature extractor of the warped point cloud does not share weights with that of the target one, would the estimation performance of the proposed method be improved? 3. Table 7 in the Appendix shows that when GMSF and the proposed method use the same architecture, the proposed method's performance improvement is marginal on F3D_o, compared with GMSF. Does such performance also occur on other datasets ( such as KITTI and Waymo-Open datasets)? It is not clear which part of the proposed method mainly contributes to the performance improvement, the architecture or the denoising process. 4. Some statements may be a bit overclaimed. The paper claims that "To the best of our knowledge, we are the first to apply a diffusion model successfully for the task of scene flow estimation.". However, as described in the main paper, a few methods have attempted to employ diffusion models for scene flow estimation [24]. The statement of "the first" may be overclaimed and can be replaced with a more proper claim.

Questions

Please refer to my comments above.

Rating

6

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

The paper provides the limitations of the proposed method, while it does not discuss the potential negative societal impact of their work.

Response to Rebuttal

Thank you authors for the rebuttal response. I have read the rebuttal and most of my concerns have been addressed. I'll stick with my "Accept" rating here.

Issues have been addressed.

I have reviewed the rebuttal and found that most of my concerns have been adequately addressed. Therefore, I will increase my rating to 'Accept'