We tackle the task of scene flow estimation from point clouds. Given a source and a target point cloud, the objective is to estimate a translation from each point in the source point cloud to the target, resulting in a 3D motion vector field. Previous dominant scene flow estimation methods require complicated coarse-to-fine or recurrent architectures as a multi-stage refinement. In contrast, we propose a significantly simpler single-scale one-shot global matching to address the problem. Our key finding is that reliable feature similarity between point pairs is essential and sufficient to estimate accurate scene flow. We thus propose to decompose the feature extraction step via a hybrid local-global-cross transformer architecture which is crucial to accurate and robust feature representations. Extensive experiments show that the proposed Global Matching Scene Flow (GMSF) sets a new state-of-the-art on multiple scene flow estimation benchmarks. On FlyingThings3D, with the presence of occlusion points, GMSF reduces the outlier percentage from the previous best performance of 27.4% to 5.6%. On KITTI Scene Flow, without any fine-tuning, our proposed method shows state-of-the-art performance. On the Waymo-Open dataset, the proposed method outperforms previous methods by a large margin. The code is available at https://github.com/ZhangYushan3/GMSF.
Paper
Similar papers
Peer review
Summary
Previous scene flow estimation methods require complicated coarse-to-fine or recurrent architectures as a multi-stage refinement. In contrast, this paper proposes a simpler single-scale one-shot global matching to address the problem. To this end, this paper decomposes the feature extraction step via a hybrid local-global-cross transformer architecture. Extensive experiments show that the proposed method achieves SOTA performance on multiple scene flow estimation benchmarks.
Strengths
1. This paper proposes Global Matching Scene Flow (GMSF) and achieves state-of-the-art performance on multiple scene flow estimation benchmarks. 2. The proposed pipeline is simple and effective. 3. The authors have provided the code in the submission.
Weaknesses
1 Some related studies have been neglected. Please compare with the previous study [cite1] in experiments. Besides, previous studies [cite2-3] need to be cited and discussed. [cite1] Lang I, Aiger D, Cole F, et al. SCOOP: Self-Supervised Correspondence and Optimization-Based Scene Flow[J]. arXiv e-prints, 2022: arXiv: 2211.14020. [cite2] Li X, Kaesemodel Pontes J, Lucey S. Neural scene flow prior[J]. Advances in Neural Information Processing Systems, 2021, 34: 7838-7851. [cite3] Li R., Zhang C., Lin G., Wang Z., Shen C.: Rigidflow: Self-supervised scene flow learning on point clouds by local rigidity prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Computer Society, Los Alamitos, CA, USA (June 2022), pp. 16959–16968. 2 Compared methods are not consistent in Table 2 and Table 3. Please clarify the reason. 3 The motivation of Eq. (11) is unclear. Please further clarify the reason why this term can be viewed as a smoothing procedure. 4 Ablation study is not enough. I suggest the authors conduct an ablation study to Eq. (13). Specifically, two version models need to be compared, i.e., model A trained with the first term and model B trained with Eq. (13). 5 The authors need to compare the FLOPs, GPU memory, and run-time of recent scene flow methods. Although the proposed method is simple and effective, the computational cost needs to be compared. 6 It seems that tokenization is too complex and redundant. Specifically, Table 5 shows that DGCGG+PT achieves almost the same performance as MLP+PT. Therefore, I think only using PT is enough, and I suggest the authors conduct experiments to report the performance of GMSF with only PT as the tokenization process. In this way, GMSF would become more efficient. 7 Whenever any abbreviations appear for the first time, it requires a full form. For example, GMSF and LiDAR.
Questions
Please see the Weaknesses. If these concerns are addressed, I am willing to improve the rating.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
N/A
Summary
The paper presents GMSF, a transformer-based method that matches dense features to estimate the scene flow from point clouds. The proposed method uses a transformer-based architecture that matches two point clouds and calculates the scene flow leveraging the cross- and self-attention modules. The paper presents experiments on FlyingThings3D and KITTI Scene Flow datasets where the proposed method consistently improves the benchmarks.
Strengths
In sum, I think the solution using cross- and self-attention mechanisms are an interesting application to solve the scene-flow problem. Although, this solution has been applied to sparse feature matching (see LoFTR), I think it is also good to see it works for the scene-flow problem. Here are the details of the strengths: S1. I find the solution an interesting application of cross- and self-attention mechanisms to solve the scene flow problem. Though in general I think the novelty is not that high since the overall idea has been used for feature matching in LoFTR CVPR 2021 [36]. S2. The clarity of the narrative is quite good. The description of the architecture and attention modules is clear. Also, the description of the losses make sense and are easy to understand. Thus, I think the paper can be reproducible.
Weaknesses
Overall, I think while the paper shows another application of attention to compute 3D scene flow, I think the paper is lacking more thorough experiments and ablations. Here are the details: W1. Missing ablations. First, the paper is not showing the performance impact of the parameters of the KNN component shown in Figure 2.I am sure this is also crucial since this allows the encoder to grab local information. How to set the KNN parameters? What is the effect of this parameter in the final performance? Second. What is the behavior of setting $\lambda$ to a different value? Why was $\lambda=0.9$ and what is the performance of the method when varying this parameter? W2. Is the KITTI Scene Flow dataset challenging enough? Since this is a dataset for autonomous driving, the motion of the vehicle is mainly planar and thus limiting the motion degrees of freedoms (only 1 for rotation, and mostly one 1 for translations, since the car moves linearly most of the time). I think this greatly simplifies the complexity of finding correspondence of any type (e.g., scene flow) in these autonomous driving datasets. W3. Lastly, I am concerned about the novelty of the approach. I think previous works have shown that attention mechanisms are useful for matching tasks in vision, and thus I struggle to find novel components or ideas in the paper. I think the paper should discuss more in depth the novelties of the paper more in detail. ---- Post Rebuttal After reading the rebuttal and discussion with the authors, my concerns have been addressed and I will increase my rating.
Questions
1. As stated in Weaknesses section, how to set the KNN parameters of the method? 2. in line 158, the paper mentions "stable tokens". However, the paper never defines what a "stable token" is.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
I think limitations are stated adequately.
Summary
This work aims to address the task of scene flow estimation for 3D point clouds. The authors propose a hybrid architecture based on local-global-cross transformers. Given as input a source and a target point cloud, first, the local transformer extracts geometric features within a patch, then the global transformer analyzes each point cloud individually using self-attention to capture the overall context, and finally the cross transformer exchanges information between the point clouds to generate the final feature representation for each point. The scene flow is predicted by performing pointwise matching with the cross similarity matrix, while occlusions are handled through a self-similarity matrix applied to the predicted scene flow. To evaluate the effectiveness of their approach, the authors conduct experiments on two benchmarks for scene flow estimation, demonstrating better performance compared to existing methods.
Strengths
- This work introduces a straightforward architecture for scene flow estimation, utilizing transformers. Without bells and whistles, the authors show that the proposed network can produce discriminative per-point features that can be robustly matched for scene flow computation. - The authors conduct extensive experiments on the FlyingThings3D and KITTI Scene Flow benchmarks in different preprocessing and occlusion settings. The results suggest that the proposed method has significant performance improvement on FlyingThings3D, while also performing competitively on KITTI Scene Flow when compared to existing methods for generalization test. - The paper is overall well written and easy to follow.
Weaknesses
- One concern for this work is its limited technical contributions: the hybrid network uses the well-established transformer architecture, while the scene flow is estimated by a common probabilistic point matching approach. One seemingly interesting proposal is the occlusion handling with the self-similarity matrix. However, it lacks in-depth explanations for why it helps with occlusion handling, and its ablation study is also missing in Sec. 4.5. - In the generalization test on KITTI-S (Tab. 3), the proposed method exhibits a performance gap compared to PT-FlowNet [8]. More detailed analysis and explanation would be helpful for gaining a better understanding of this discrepancy. - For the ablation study in Tab. 4, it is unclear whether the performance saturates with eight global-cross transformer layers, and whether more layers would be beneficial or not. To provide a comprehensive assessment, it would be good to include comparisons of runtime and memory usage, as the backbone is built upon transformers, which may not scale well with more points, and the paper mentions that input point clouds are resampled to 8K points. - Minor: L267, “Although we don’t” => Note that we do not
Questions
Please see the Weaknesses section.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
The authors discussed limitations at the end of the paper.
Summary
This paper proposes a hybrid local-global-cross transformer scene flow estimation model, achieving the state-of-the-art results on FlyingThings3D and KITTI Scene Flow datasets.
Strengths
1. This work has a mature model design based on Transformers, achieving superior results on the Flyingthings3D and KITTI Scene flow datasets. 2. The structure and writing of this paper do not have major issues.
Weaknesses
1. Scene flow estimation has been developed for many years, and the autonomous driving industry has already introduced Occupancy and Flow Prediction techniques. Researchers should not be limited to designing a toy model solely to maximize the benchmarks on FlyingThings3D and KITTI Scene Flow. These two datasets have dense point clouds and clear correspondences, which are far from practical applications. For this paper, I hope the authors can conduct more experiments on the Waymo motion data (https://waymo.com/open/data/motion/). 2. Undoubtedly, Transformers will bring more powerful feature modeling capabilities and higher latency. How does the latency of this method compare to competitors? Please add this item to the main experimental table.
Questions
Please refer to the weaknesses section, refute views or answer questions or improve the paper.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
The limitations has been discussed.
Summary
The paper proposes GMSF for scene flow estimation from point clouds. As far as the authors are aware, GMSF is the first to address scene flow estimation with global matching - GMSF is formulated as a single-scale one-shot global matching. GMSF incorporates a novel local-global-cross transformer architecture to extract high-quality feature representation, to finally compute the scene flow between point clouds via global matching. GMSF outperforms existing methods on F3D_c, F3D_o, F3D_s and KITTI_o, while performing competitively on KITTI_s. The ablative results show that increasing the number of global-cross transformer layers - thus increasing the capacity - is beneficial to the performance, and that the presence of local information is crucial for the performance of GMSF.
Strengths
* The paper is overall well written and easy to follow. * The proposed architecture, and the motivation behind it, is intuitive and novel. While the architectural novelty itself is not entirely new (e.g., the sequence of local and global attention has first been introduced by SuperGlue[1] and LoFTR[2] for 2D image matching, and has been applied to 3D point cloud registration through methods including CoFiNet[3] and GeoTransformer[4]), its application to the task of scene flow seems conceptually new. * Strong performances on the standard benchmarks of scene flow estimation. [1] PE Sarlin et al., SuperGlue: Learning Feature Matching with Graph Neural Networks, CVPR 2020 [2] J Sun et al., LoFTR: Detector-Free Local Feature Matching with Transformers, CVPR 2021 [3] H Yu et al., CoFiNet: Reliable Coarse-to-fine Correspondences for Robust Point Cloud Registration, NeurIPS 2021 [4] Z Qin et al., Geometric Transformer for Fast and Robust Point Cloud Registration, CVPR 2022
Weaknesses
* Insufficient ablative experiments. What if the number of transformer layers increase to a number higher than 8? The given results show that the performance improves gracefully with the number of layers, and it naturally leaves the question to 'until how much'. Also, what if the global transformer and cross transformers are decoupled, such that they can have varying number of layers? * Lack of latency and computation analyses. The authors emphasize that GMSF is a single-scale, one-shot method; then how does it compare to existing methods in terms of latency and computation (FLOPs, memory)? * Lack of analysis. How does incorporating global-cross transformer layers improve the performance, and how does incorporating **more** layers **further** improve the performance? This has been partially answered by Table 4, but a visual comparison / analysis would be more convincing. * Lack of mention of 3D point cloud registration methods. While the task at hand is different, the architectural design and motivation is closely related to 3D point cloud registration, which I believe is therefore worth mentioning in the related work section. Also, it might be a bit of an overstatement to mention that GMSF is the 'first' to address scene flow estimation with global matching, as scene flow estimation and point cloud registration are seemingly closely related tasks .
Questions
Please refer to the weaknesses section. The motivation and the proposed method are intuitive and novel, but the design choice of GMSF should be better substantiated, with included analyses for clarity.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
The authors have addressed the limitations of GMSF in the paper.
Thank you for your response. I still have the following concerns about GMSF. (1) The comparison with SCOOP [cite1]. Because the authors have provided codes, I trained both SCOOP and GMSF by myself. However, SCOOP is faster, data-efficient (one-tenth of the whole training set), and easy to be trained (less than 1h with a single 3090 trained on FT3D). In contrast, the GPU requirement of GMSF is too large, and I cannot train the GMSF on a GeForce RTX 3090 (24G) even with bs = 1. More crucially, SCOOP is an unsupervised method without ground truth flows. Please clarify the strength of GMSF. (2) About the tokenization process. I am confused about the table in the answer to Q6. Which row of the table represents the performance of GMSF with only PT as the tokenization process? According to Concern (1), I strongly suggest the authors simplify the architecture of GMSF. Otherwise, it is impossible to use GMSF for real-world applications.
Thanks for the reply and for the effort of running the experiments.
Thanks for the reply and for the effort of running the experiments. Q1: The strength of GMSF. Methods, such as SCOOP, based on prior information and runtime optimization are usually easier to train but more time-consuming during inference ($10\times$–$100\times$ slower, refer to [1]). Therefore, when there is a requirement on inference time, runtime optimization-based methods are not suitable. For example, SCOOP [2] achieves the best performance with an inference time of \~20s, Neural Prior [1] with an inference time of \~40s (see Figure 7. in SCOOP). This is around two orders of magnitude higher than our inference time (371.8ms). Moreover, during inference, our method requires 5GB of memory, and can thus run on a Geforce RTX 3090 or lower end graphics cards. Q2: Table in the answer to Q6. The result of GMSF with only PT is not in the table. The table is with DGCNN+PT, but the dimensions of the feature space are reduced from 128 to 64 in the first row. As mentioned, the function of DGCNN and MLP is to project the 3D coordinate into the high-dimensional feature space. Without them and only applying PT on the 3D coordinate would result in a lack of capacity of the model. The table shows a performance drop with fewer feature dimensions, and presumably a further simplification could result in a larger performance drop. (We have tried training GMSF with only PT, but this has problems with convergence.) However, another way to simplify our model is to reduce the number of global-cross transformer layers (see Table 4 in the paper). The table shows that our proposed method has a good performance even with only 4 global-cross transformer layers. (We have tried with global-cross transformer layers = 4, batch size = 1. It converges well with 14.5GB memory usage.) Besides, since most of the applicability in real scenarios is based on inference, this makes inference time arguably the most important factor in the real-world applicability of a method. We are on par with the other state-of-the-art methods (**two orders of magnitude faster than SCOOP**). Moreover, during inference, our method only requires 5GB of memory, therefore, it can even be run on lower end consumer graphics cards. We hope that could help with real-world applications. References [1] Li X, Kaesemodel Pontes J, Lucey S. Neural scene flow prior[J]. NeurIPS2021. [2] Lang I, Aiger D, Cole F, et al. Scoop: Self-supervised correspondence and optimization-based scene flow[C]//CVPR2023.
Thank you for your response. I partially agree with the authors' opinion that inference time may be the most important factor in real-world applications. Human annotations/labels requirements, generalization to out-of-distribution data, and computational complexity are all important factors. However, it is impossible for one paper to solve all the problems. In this way, I suggest the authors cite and discuss the recent paper [cite1] in the revision (no need to do this during the discussion phase). GMSF is effective and achieves SOTA performances among supervised scene flow methods. Although GMSF still has some drawbacks, such as high training difficulty and the supervised scheme, the contribution and novelty of GMSF are strong enough to be accepted. I think the added experiments should be included in the revision to further strengthen the paper. I am willing to raise my rating. [cite1] Chodosh N, Ramanan D, Lucey S. Re-Evaluating LiDAR Scene Flow for Autonomous Driving[J]. arXiv preprint arXiv:2304.02150, 2023.
RE: Rebuttal by Authors
Thanks for the clarifications. A few more questions: Q1a: It seems like the parameter $k$ has marginal impact, and that k=16 "best" performance may be within the margin of error. Any insight as to why $k$ has a marginal impact on the method? Q2: The reason I raised this concern is to understand the generalization of the proposed method. Although the results from FT3D partially satisfy my question, the FT3D dataset is synthetic and may pose a synthetic-real gap.
Thanks for the reply. Q1: Marginal impact of $k$. We believe that the marginal impact of $k$ is due to the design choice of our tokenization process. The tokenization consists of a DGCNN and a Point Transformer, both of which encode local information. Table 5 in the paper shows that as long as there is some local information encoded, the performance remains competitive (EPE=0.049 for DGCNN+PT, EPE=0.055 for DGCNN). Only changing the $k$ parameter in the Point Transformer does not affect the local information encoded by DGCNN, thus has a limited impact. Moreover, Local information of only 4 points already includes features such as curvature. These features become more distinct the more neighboring points are considered. But there is a natural limit when they don't add any more information. Q2: Performance on the real dataset. Since the annotation of real data is very expensive in scene flow estimation, synthetic datasets are usually employed during training [1, 2]. This is also the case for optical flow estimation [3, 4], where the same FT3D dataset is used during training and shows a good generalization performance on the KITTI dataset. However, we agree that when training only on synthetic datasets, the synthetic-to-real-gap should be carefully analyzed. According to the suggestion from Reviewer QN7Z, we extend our experiments to the larger and more challenging Waymo-open autonomous driving dataset that contains 798 training and 202 validation sequences (each 20 seconds with 10Hz point clouds data) with more complex scenes compared to the KITTI dataset. The results in comparison with state-of-the-art methods are given in the following table. We outperform the state of the art by a large margin. We hope this could help with your concern about the applicability of our method in real-world scenarios. | method | $EPE_{3D}\downarrow$ | $ACC_{S}\uparrow$ | $ACC_{R}\uparrow$ | $Outliers\downarrow$ | |-----------|-----------|-----------|-----------|-----------| | FlowNet3D[1] | 0.225 | 23.0 | 48.6 | 77.9 | | PointPWC[5] | 0.307 | 10.3 | 23.1 | 78.6 | | FESTA[6] | 0.223 | 24.5 | 27.2 | 76.5 | | FH-Net[7] | 0.175 | 35.8 | 67.4 | 60.3 | | **GMSF** | 0.086 | 73.9 | 84.7 | 43.9 | References: [1] Liu X, Qi C R, Guibas L J. Flownet3d: Learning scene flow in 3d point clouds[C]//CVPR2019. [2] Gu X, Wang Y, Wu C, et al. Hplflownet: Hierarchical permutohedral lattice flownet for scene flow estimation on large-scale point clouds[C]//CVPR2019. [3] Teed Z, Deng J. Raft: Recurrent all-pairs field transforms for optical flow[C]//ECCV2020. [4] Jiang S, Campbell D, Lu Y, et al. Learning to estimate hidden motions with global motion aggregation[C]//ICCV2021. [5] Wu W, Wang Z Y, Li Z, et al. Pointpwc-net: Cost volume on point clouds for (self-) supervised scene flow estimation[C]//ECCV2020. [6] Wang H, Pang J, Lodhi M A, et al. Festa: Flow estimation via spatial-temporal attention for scene point clouds[C]//CVPR2021. [7] Ding L, Dong S, Xu T, et al. Fh-net: A fast hierarchical network for scene flow estimation on real-world point clouds[C]//ECCV2022.
Comments on Authors' Rebuttal
I appreciate the authors' effort in addressing my questions in the rebuttal. After reading through the review comments from other reviewers, although I am still concerned about the technical novelty, as commented by the fellow reviewers, I think this work can serve a stepping stone for future works on scene flow given its comprehensive and improved results. It would be great if the authors can release the code and trained models to benefit the community. As of now, I would like to keep my rating.
Thank you authors for the detailed responses to my concerns. They seem to have been well addressed. I still appreciate the conceptual novelty and the strong performance of the paper, but am still concerned about the limited technical novelty of the paper. I would like to maintain my current rating of borderline accept.
Has the rebuttal addressed your concerns
Dear Reviewer QN7Z, Could you please read the author rebuttal and acknowledge if your concerns have been addressed? The discussion period will end very soon on Monday, August 21. Thank you for your time in reviewing this submission! Best, AC
Sorry for my late reply. If the method can bring such a significant improvement, I will change the score to positive. Please merge the new waymo experiment into the camera ready version. Thanks.
Decision
Accept (poster)