ZeroFlow: Scalable Scene Flow via Distillation

Thank you for your review. # Idea lacks novelty Michael Black’s reviewer guidelines [1] eloquently argue that "the simplicity of an idea is often confused with a lack of novelty when exactly the opposite is often true. Such reviews lead researchers to make an idea appear more complex so that reviewers will find it of higher value." Indeed, our novelty comes from the fact that our simple and post-hoc obvious pipeline produces state-of-the-art results. To quote q16v: "This distillation idea is very simple, but previous work on point-cloud-based scene flow methods hasn't tried the idea yet. The paper demonstrates that this simple idea is working and demonstrates that it's scalable by using large-scale diverse unlabeled data." This result also produces "valuable insights to the autonomous vehicle industry" (pGhB). # This paper would be better suited for ICRA or WACV, as they tend to focus on more application and system-oriented research Our paper provides non-trivial, fundamental insights: our straight-forward pipeline shows that simply training a supervised model with imperfect pseudo-labels can *exceed* the performance of perfect human labels given sufficient unlabeled data. We think this is itself surprising, but we also think it has significant implications for the problem of scene flow estimation in general: *point cloud quantity and diversity is more important than perfect flow label quality for training feedforward scene flow estimators*. We translate this into an empirical scaling law that describes the log-log relationship between dataset size and final Threeway EPE performance. We also present a novel scene flow estimation analysis technique. To our knowledge, the residual plots in Section 4.4 are the first attempt at visualizing the residual *distribution* of scene flow estimators. We think these plots provide useful insights to practitioners and researchers. For example, open world object extraction [2] must be able to threshold and cluster motion vectors together to extract the entire object. Understanding the *distribution of flow vectors* is essential for identifying extraction heuristics. . Furthermore, we provide extensive scientific inquiries on the impact of not only method component choices (e.g. Teacher quality — Section 4.5), but of the data itself (e.g. Dataset diversity — Section 4.3). These inquiries provide important insights to practitioners and researchers on what are fruitful directions to explore. We believe this makes our paper of interest to the ICLR community looking at the "applications to robotics, autonomy, planning" primary area. # Numbers on dataset point cloud sizes We provide plots of the full point cloud size distribution for AV2 and Waymo datasets after ground point removal and region clipping (102.4m x 102.4m area around the ego vehicle) in the Supplemental, Section A. The average number of points per point cloud is 52,871.6 points for AV2 and 79,327.8 points for Waymo Open. # Removing Algorithm 1 Thank you for pointing this out. We will address this for the camera ready. [1] Michael Black. Novelty in science: A guide to reviewers.

Reviewer pGhB, did the authors' rebuttal had addressed your concerns and questions? Please reply and post your final decision. AC

Thank you for your review. # Novelty Michael Black’s reviewer guidelines [1] eloquently argue that "the simplicity of an idea is often confused with a lack of novelty when exactly the opposite is often true. Such reviews lead researchers to make an idea appear more complex so that reviewers will find it of higher value." Indeed, our novelty comes from the fact that our simple and post-hoc obvious pipeline produces state-of-the-art results. To quote q16v: "This distillation idea is very simple, but previous work on point-cloud-based scene flow methods hasn't tried the idea yet. The paper demonstrates that this simple idea is working and demonstrates that it's scalable by using large-scale diverse unlabeled data." This result also produces "valuable insights to the autonomous vehicle industry" (pGhB). # Performance of our system is the result of its components Our performance cannot be strictly attributed to the components plugged into our *Scene Flow via Distillation* framework — we benchmark ZeroFlow and outperform both NSPF and FastFlow3D. But, broadly, we agree that we benefit from high quality components! Indeed, a feature of our *Scene Flow via Distillation* pipeline is, given a better test-time optimization method, a better supervised scene flow network architecture, or more unlabeled data, it can leverage these component improvements to produce an even better final method *for free*. # More analysis of of the combined strengths of optimization-based and feed-forward methods We refer the reviewer to Section 3.1, which analyzes the combined strengths of optimization-based and feed forward methods. Optimization methods produce high quality flow, but have high computational cost, whereas feed-forward methods are fast but limited by the scale of their supervised labels. Combining these two techniques gives us a formula for scaling that allows our combined method to exceed the performance of its components and achieve state-of-the-art results. We would gladly appreciate suggestions for further analysis. # Waymo Open (Table 2) results not competitive ZeroFlow provides a label-free, scalable training pipeline. We expect that, relative to human supervision *on the same data*, ZeroFlow's pseudolabeling approach will perform worse. This is reflected in ZeroFlow's performance on Argoverse 2 (Table 1); ZeroFlow 1x performs worse than its supervised counterpart, FastFlow3D. However, when scaling up training to include the unannotated Argoverse 2 LiDAR dataset, ZeroFlow exhibits meaningfully superior performance compared to the human supervised counterpart (ZeroFlow 3X vs FastFlow3D). This same pattern is also reflected in Table 2 for Waymo Open. Waymo Open does not provide any additional unlabeled data beyond their annotated dataset, preventing us from scaling up ZeroFlow further; however, our performance results on the data we do have are actually quite positive. Supervised FastFlow3D has a very similar Threeway EPE on both datasets, while ZeroFlow 1x on Waymo Open is actually closer in performance to FastFlow3D than on AV2 where we successfully demonstrated that more raw data produces state-of-the-art performance. # This is a technical report not a research paper; mostly engineering Our paper provides non-trivial, fundamental insights: our straight-forward pipeline shows that simply training a supervised model with imperfect pseudo-labels can *exceed* the performance of perfect human labels given sufficient unlabeled data. We think this is itself surprising, but we also think it has significant implications for the problem of scene flow estimation in general: *point cloud quantity and diversity is more important than perfect flow label quality for training feedforward scene flow estimators*. We translate this into an empirical scaling law that describes the log-log relationship between dataset size and final Threeway EPE performance. We also present a novel scene flow estimation analysis technique. To our knowledge, the residual plots in Section 4.4 are the first attempt at visualizing the residual *distribution* of scene flow estimators. We think these plots provide useful insights to practitioners and researchers. For example, open world object extraction [2] must be able to threshold and cluster motion vectors together to extract the entire object. Understanding the *distribution of flow vectors* is essential for identifying extraction heuristics. . Furthermore, we provide extensive scientific inquiries on the impact of not only method component choices (e.g. Teacher quality — Section 4.5), but of the data itself (e.g. Dataset diversity — Section 4.3). These inquiries provide important insights to practitioners and researchers on what are fruitful directions to explore. [1] Michael Black. Novelty in science: A guide to reviewers. [2] Najibi et al. Motion Inspired Unsupervised Perception and Prediction in Autonomous Driving. European Conference on Computer Vision. ECCV 2022.

Reviewer E12X, did the authors' rebuttal had addressed your concerns? Please reply to the author and post the final decision. AC

Thank you for your review. As mentioned in the abstract, we intend to release all of our source code along with the pseudolabels we used to train ZeroFlow. # Why does ZeroFlow's scaled up performance surpass the teacher's performance? ZeroFlow's student-teacher distillation setup enables the student to surpass the teacher because neural networks are good at learning from noisy labels [1]. As we show in Figure 5, as ZeroFlow's pseudolabeled dataset is scaled up (1x, 3x), the variance of its error distribution shrinks because the student gets better at predicting the *expected value* of the NSFP motion estimates conditioned on the observation of motion. This lower variance expected value estimation leads to lower Threeway EPE estimates compared to even NSFP itself. # Pseudolabel performance in Table 1 As shown in Table 1, NSFP (our pseudolabeler) achieves 0.068 EPE. # Why does FastFlow3D outperform ZeroFlow when the dataset size is less than 100%? FastFlow3D is trained using ground truth flow provided via human annotations. ZeroFlow is trained using pseudolabel flow provided via NSFP. NSFP is an imperfect pseudolabeler, so even though ZeroFlow and FastFlow3D share the same network architecture and see the same point clouds, ZeroFlow's supervision is from inferior quality pseudolabels. Therefore, performance is expected to be worse than its human supervised counterpart. # Faster teacher method Thank you for this reference! We will look into this as a faster teacher! [1] Li et al. Learning from Noisy Labels with Distillation. ICCV 2017.

Reviewer q16v, while the author had posted the rebuttal, please reply if you still have any concerns, or post your final decision based on the reviews, rebuttals and the revision. AC