Recent advances in machine learning have shown that Reinforcement Learning from Human Feedback (RLHF) can improve machine learning models and align them with human preferences. Although very successful for Large Language Models (LLMs), these advancements have not had a comparable impact in research for autonomous vehicles -- where alignment with human expectations can be imperative. In this paper, we propose to adapt similar RL-based methods to unsupervised object discovery, i.e. learning to detect objects from LiDAR points without any training labels. Instead of labels, we use simple heuristics to mimic human feedback. More explicitly, we combine multiple heuristics into a simple reward function that positively correlates its score with bounding box accuracy, \ie, boxes containing objects are scored higher than those without. We start from the detector's own predictions to explore the space and reinforce boxes with high rewards through gradient updates. Empirically, we demonstrate that our approach is not only more accurate, but also orders of magnitudes faster to train compared to prior works on object discovery.
Paper
Similar papers
Peer review
Summary
The paper addresses the research question of unsupervised 3D object location detection from LIDAR data in autonomous driving scenes. The proposed method, DRIFT, improves upon the MODEST baseline by incorporating heuristics for judging objectness likelihood and using them as rewards within a reinforcement learning framework. DRIFT is evaluated on the LYFT and Ithaca benchmarks, demonstrating enhanced performance and training efficiency compared to the MODEST baseline. **Before rebuttal**: The paper presents a promising contribution and could be a valuable addition to NeurIPS this year. However, since my familiarity lies more in the domain of object detection from 2D scenes, I may not be able to provide detailed insights into the previous work in the LIDAR domain.
Strengths
**S1.** The paper is well-written, well-executed, and effectively addresses the under-explored problem of unsupervised 3D object location detection in autonomous driving scenes. The proposed method, DRIFT, demonstrates the potential for real-life applications. **S2.** The proposed idea is well-grounded and logical. The incorporation of studied priors based on shape, size, and location to judge objectness likelihood aligns with human intuition. The utilization of a feedback loop through reward-based fine-tuning to improve model performance over time is a sensible approach. **S3.** The reported improvement achieved by the authors is significant. Despite training for only 30 epochs (compared to MODEST's 600 epochs), DRIFT outperforms the strong baseline on both datasets. Although there is still a noticeable gap when compared to supervised methods, the results demonstrate promising progress.
Weaknesses
**W1.** It would have strengthened the paper to provide a context within the well-studied objectness literature [1-2]. The concept of objectness aims to identify low-level, generic cues that distinguish foreground regions from the background, such as edge distributions, boundary textures, and likely object size, shape, and location. The authors' goal aligns with this objective, albeit from a different modality (LIDAR). Drawing inspiration and techniques from the objectness literature could have been insightful. **W2.** An explanation of how the authors generated the shape templates (priors, prototypes) would have been beneficial. It is unclear how these templates were derived or selected. **W3.** Given the reliance on multiple heuristics in the method (which is reasonable and sound), it would have been valuable to investigate cross-dataset generalization, specifically from LYFT to Ithaca. Understanding how the measured statistics change across different domains and driving scenes would provide insights into the method's robustness and applicability. [1] BING: Binarized normed gradients for objectness estimation at 300fps, https://mmcheng.net/mftp/Papers/ObjectnessBING.pdf [2] Survey and Performance Analysis of Deep Learning Based Object Detection in Challenging Environments, https://www.mdpi.com/1424-8220/21/15/5116
Questions
See the weaknesses. More like a suggestion: Please state from the very start that your goal is to find generic (foreground vs. background) object regions, and NOT categorization. This maybe confusing to some readers like me, that when you use the term "Object Discovery", I look for a semanic categorization component as well.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Limitations are presented.
Summary
The proposed DRIFT framework is an approach to realize object discovery without labels. DRIFT first extracts foreground proposals based on the PP-score method, and then leverages common-sense heuristics including shape prior, box alignment, and background point filtering, to reward proposed boxes. Reinforcement-learning-based optimization is adopted, maximizing the reward function in a local space. The experimental results demonstrate the superiority of DRIFT over prior self-training methods in terms of efficiency and generalizability.
Strengths
- The framework for 3D object discovery that simplifies common-sense heuristics as Gaussian-based reward signals to fine-tune the detector is interesting, straightforward, and relatively novel. Since the object discovery paradigm does not incorporate pre-defined class labels, it is promising in out-of-domain object perception tasks. - Experiments are basically comprehensive, with ablations on key reward components and plenty of hyperparameters. - The paper is well-written, and the ideas are presented clearly, making it easy to follow the authors' arguments and understand their contributions.
Weaknesses
- The authors mainly compare DRIFT with proposal-based (PP-score) baselines. - A missing related work for comparison: [a]. [b] is a concurrent work, but is valuable to discuss it though a direct comparison is not feasible. - The PP-score method requires multiple traversals to get a superior performance which could limit the generalizability in real-world applications. This helps DRIFT to achieve a closer out-of-domain supervised detector trained on KITTI in the Ithaca365 dataset. But in normal datasets without many traversals, DRIFT actually has a large gap compared to the out-of-domain supervised detector. - Current heuristics behave poorly in pedestrians, as shown in the per-class BEV mAP of the pedestrian class in Fig. 6 & Supp-Fig. 1. Heuristics need careful designs and more sophisticated ones are probably necessary to achieve satisfactory results on all categories of objects. - The reviewer is curious about the training efficiency. Any more insights about why the proposed DRIFT converges much faster than self-training and MODEST? - Any more insights about the large StD. in Table 1? - Misc: Fig. 1 is not referenced in the main body. > [a] Motion Inspired Unsupervised Perception and Prediction in Autonomous Driving. ECCV 2022. > > [b] Towards Unsupervised Object Detection from LiDAR Point Clouds. CVPR 2023.
Questions
Please see weaknesses above.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
The authors have discussed limitations at the end of the paper. The discussion is valuable.
Summary
This paper proposes a new reinforcement-learning-based framework for unsupervised 3D object detection that uses these common-sense heuristics directly as a reward signal. Avoid handcrafting training examples for each object detector. Furthermore, under the premise of greatly accelerating the convergence speed of the model, this method improves the experimental effect.
Strengths
1. The proposed reinforcement learning-based framework without encoding heuristics into differentiable loss functions and avoids the need to hand-engineer training paradigm. 2. The method proposed by the author has a fast convergence speed and a great algorithm implementation value. 3. The experiments in this article are relatively sufficient, and the charts are easy for readers to understand.
Weaknesses
1. The content arrangement of this article does not seem to be very convenient to read and there are some careless quoting errors that refer to Questions. 2. The author should provide further explanation or derivation in the formula part of the article.
Questions
1. In line 40, can it be understood that the method proposed in this article is to perform finetune on the pre-trained MODEST through a reward-based method? If the answer is yes, is the time comparison in Figure 1 meaningful or fair? Moreover, LLM is mentioned more than once in the article, but this article does not seem to use a specific LLM model. I think this may cause some misunderstandings for readers at the beginning. 2. The method proposed in this paper is more like a specific RAFT method in unsupervised object detection, and the novelty may be limited. 3. The training steps mentioned in Line 56. As far as I know, many self-training methods can achieve the same goal, please tell us your advantages. 4. Lines 126 to 132 in the article seem to give the boxes from the detector a priori information or heuristic constraints from the real world. You are modeling here by a mixture of Gaussian distributions, how to determine mixture weights? In line 115 section. What happens if you use a mixture factor greater than 4? It seems that the model only focuses on objects of these sizes. In line 131, what happens if you don't use the scale operation on it? please give an example. 5. Is Figure 3 drawn from a real dataset? If so, please describe which data were used in the supplementary material. Can this be understood as prior information derived from daily data? 6. The paragraph starting on line 140 means that by scaling, more points are included in the box, this seems to serve the same purpose as jittering mentioned in Sec 3.2 or Intro. What is the difference between them? 7. The sensitivity analysis of mu_{scale} in line 156 should be Tab5. 8. The ablation experiment of Tab4 should complement the other 3 groups and did not analyze the reason for this phenomenon, especially when the effect gap is huge. Because I want to know how the three components of the proposed Reward function affect the detection performance respectively 9. Please confirm the experimental code for reproducing the MODEST in Tables 1 and 2. Whether there is a P2 filtering result seems to be far from the original MODEST paper.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
One limitation is that this paper uses MODEST as the baseline. If the number of timestamps is not large, continuing to use the p2 score may aggravate the impact of noise on the model. This is a direction that needs to be addressed. Another limitation is that since it is unsupervised 3D object detection, not only static mobile objects should be detected, but all static objects should be detected, which may be beneficial to downstream tasks.
Summary
This paper proposes DRIFT, a novel reward fine-tuning method for unsupervised object discovery with point cloud input. Specifically, three reward methods are proposed to identify good bounding boxes. First, shape prior reward prefers bounding boxes with similar sizes to the prototypes. Second, an alignment reward gives high scores to boxes that have most of the LiDAR points near the box edge. Third, a filter reward follows the spirit that a good box should contain more dynamic points than background points. By using these reward methods for refining the object detector, DRIFT achieves more accurate object discovery, and it also converges much faster than the previous method.
Strengths
(1) The proposed reward fine-tuning method is well-motivated and improves object discovery accuracy compared to the previous work. (2) In addition to improving the detection performance, the proposed method can also greatly improve the training speed. (3) The expression of the paper is clear, and the figures are intuitive.
Weaknesses
(1) Although intuitive, I think the prior that most points fall on the lateral surfaces of an object is too absolute, e.g., a lot of LiDAR points will fall on the hood and front window of the oncoming vehicles beside lateral surfaces. Apart from this, using Gaussian distribution as an approximation is also not accurate, because few points fall out of the box edge, as shown in Figure 3. (2) I think a missing ablation is the contribution of the exploration strategy to the final detection performance, similar to the ablation on the reward methods in Table 4.
Questions
(1) Authors should provide complete experiment results and refine their experimental analysis correspondingly, as some of the experiments are still running at the time of submission (L234 & L258). (2) In L224, the authors say they include evaluation results with IoU at 0.25 in the supplementary, but I don't see these results. Authors should add these results and explain why they don't provide these results in the main paper.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors provide limitations.
Summary
This work contributes to unsupervised object discovery in LIDAR point clouds. It defines a few typical properties of customary bounding boxes in LIDAR point clouds and then develops Rewards for a Reinforcement Learning algorithm to learn this, lacking a gradient for direct optimization. It is based on Persistency Prior scores defined by the MODEST approach but adds several steps to improve bounding box selection. Furthermore they ablate many factors, e.g. the impact of enforcing a certain shape on bounding boxes. While they can only compare on a limited amount of epochs due to time constraints, possibly due to the plenty ablation studies, they show they outperform the existing approach in that domain.
Strengths
The paper improves performance by a large margin in a very novel research area. Even if the approach itself may have limited use, it is straightforward to see the value e.g. in combination with a pre-trained traffic participant classifier. The easy and intuitive explanation of the reward shaping is very clear. Figure 2 helps to get a very quick insight into the idea. The whole paper does a good job at combining intuitive explanations with mathematical formulations. The approach has many individual steps and the pseudo-algorithm helps understanding. The used Lyft Level 5 Perception dataset is a good choice for comparing with the state of the art. The ablation studies are suitable to show that all used steps together achieve the performance. All figure captions guide the reader nicely towards the main message that figure should convey.
Weaknesses
It seems this work takes many ideas from MODEST so the idea of using an object detector and defining common sense properties is not novel to this work. It needs to be read with that contribution offset in mind. The title is maybe a bit boasting. Teaching cars to see sounds like solving most if not all perceptual problems while this approach, coming from unsupervised object detection, only separates moving objects from background. Having still a bit to train is not optimal but understandable. The paper feels a bit rough around the edges, the DRIFT abbreviation is explained twice. The approach is a bit complex with many parameters to tune. Even though the ablation studies show the value of all individual components, it is hard to judge how much hyperparameter optimization vs. unsupervised modeling performance is present.
Questions
2 Related Works: 3D Object Detection: I would disagree it is ideal to train detectors in an unsupervised way. Maybe ideal could be a combination to achieve best performance at reasonable costs while being able to assign actual labels (car, pedestrian, etc.). Maybe a statement like that could be just removed. How robust is the algorithm to the amount of traversals? This is not touched at all unless I overlooked it. Jittered boxes and then NMS has strong similarity to region proposals in approaches like Mask R-CNN. Maybe it could be good comparing this historic approach in the Related Work. What is the impact of the quality of the first pre-trained detector? If there is a false negative detection, how can this algorithm improve a non-existing bounding box? In Reward Finetuning for Model Alignment, what are "human values"? Do the models output unexpected or wrong output or are there ethical considerations? The term seems confusing. How have the lambda factors and mu_scale and sigma_scale been found? Similarly how was the scale of the scaled up box selected when designing the Alignment Reward? Since the approach is supposed to find objects unsupervised, hyperparameter tuning and then manually checking the results by a human who looks at the bounding box detections in the point cloud may improve the results but not in a fair way. It goes against the idea of unsupervised object detection. In 3.2 what is meant by "In effect, this encourages all non-persistent points ... to propose boxes". How do points propose boxes? This is not like Yolo where each grid cell is associated with a fixed number of region proposals right? Couldn't this approach potentially be expanded to actually label Car, Pedestrian, Cyclist and Truck based on the shape priors? It would be worthwhile investigating the impact of false negatives to understand the impact of the approach vs. the failure of the first step of bounding box proposal. The authors did compare different ways of box predictions but that is different from quantifying the performance. Knowing how many false negatives are produced by the first step would already give some insight into this. Or is that the No Finetuning case?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
There is only one comparable work which is an unavoidable limit. The authors solved this by also comparing against other tasks, i.e. the supervised case. Due to the deadline DRIFT was not trained to convergence, however it seems to outperform it's competitor already. For the camera ready final numbers should be entered. In their limitations the authors address most of the concerns in Weaknesses.
Post-rebuttal Comment
Thanks for the feedback. **[Re: Large Std.]** My intent was to ask the large Std. in Table 3 compared with other models in the main paper. The number in my original question was a typo. Sorry for the mistake. Do you have more insights about the large StD. in Table 3?
Insights into Table 3 StD.
In general, bad boxes consistently yield low rewards, as seen in the low mean and low StD. of the "rand boxes" row in Table 3. Good boxes tend to get high rewards on average, but the rewards may vary across different boxes because the signal used to compute the rewards can be noisy though still correlated with box quality. For instance, one part of our reward function, dynamic/static point counts, varies depending on the proximity of the box to the LiDAR sensor, since LiDAR point density decreases with range. As another example, the P2-score we used as an approximate foreground segmentation is a noisy heuristic for the dynamic points. Thus while the rewards are correlated with actual box quality (as shown by the high mean reward for the highest quality boxes, i.e., the ground truth), the correlation is not perfect. That said, note that this large StD. has a minimal impact on DRIFT, as the top *k*% filtering retains mostly good boxes, allowing the detector to learn to identify dynamic objects from them.
Read the comments of the authors and other reviewers comments. I am content with the responses.
Dear reviewers, We thank you so much for your time and effort, and would be happy to answer any further questions you may have before the discussion period ends. Please let us know if any issues remain and/or if there are any additional clarifications we can provide. If you are satisfied with our rebuttal, we would appreciate it if you could reconsider your score. Best regards, Authors
I appreciate the efforts the authors spent on their rebuttal! It solves some of my concerns. But I have two other questions. 1. As mentioned by reviewer iC9C, 'Since the approach is supposed to find objects unsupervised, hyperparameter tuning and then manually checking the results by a human who looks at the bounding box detections in the point cloud may improve the results but not in a fair way. It goes against the idea of unsupervised object detection.' While your answer is that these parameters are very robust to other datasets, this seems to be at the expense of learning from the data, can you give another explanation? 2. The scoring function designed in Sec3.1 seems too heuristic. In 2d unsupervised object detection, there are some works like[1] which leverage VLMs to generate or filter pseudo labels, can you refer to these ideas to further improve the effect of the model or reduce the complexity of the model? But in my opinion, I think the novelty of this paper is still limited, and I insist on my points for now. [1] Zhao, Shiyu, et al. "Exploiting unlabeled data with vision and language models for object detection." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.
Thank you for your response.
We respectfully disagree that this method lacks novelty; it’s a compellingly simple, effective, and robust framework, as shown in the variety of ablations both in the paper and in the subsequent rebuttal. However, we agree that there are *ways that the method can be expanded* within this framework. Part of our contribution lies in the general framework that can leverage functions of any type, differentiable or not, in an efficient manner, and bridges the concept of exploration mechanism, common for the field of RL, successfully into object discovery. We thank the reviewer for pointing to another way we can extend the reward and exploration function to incorporate additional modes of information such as language [1]. While out of scope for this work, our future works aim to explore reward/exploration functions that are appropriate for such settings (L307-308). However, our current contribution re-thinks the problem of object discovery in a fundamentally new way, and in doing so, yields incredible performance gains. Indeed, reviewers iC9C, vhou, and Baoq consider the method to be novel and a valuable contribution, and we would be happy if reviewer dARk can reconsider their review under this light. > Hyperparameter tuning, domain knowledge, and learning from data. We base the selection of many of our hyperparameters on human domain knowledge, which does not necessarily need direct learning from the data. For instance, we can readily obtain dimensions of transportation tools from online sources [2], and estimate human body shapes using health statistics reports from the CDC [3]. This strategic application of domain knowledge enables us to scale to large amounts of scenes and does not require human involvement for each individual scene. We contend that to do unsupervised learning, we must assume some knowledge, either from domain knowledge, learned from human labels, or distilled from other datasets. In this work, we assumed access to simple and easily accessible domain knowledge and believe it is the most scalable solution. > Leveraging VLMs to generate or filter pseudo labels To clarify, the scoring function is the objective we wish to optimize. Proposals (i.e., pseudo-labels) are guided by exploration to encourage identifying boxes that can improve the scores (i.e., reward) that are obtained. Better proposals will help get the model to better regions quicker, thus increasing the efficiency of the method, but ultimately the objective of the method is to maximize the reward. Under our framework, if the aim is to improve the reward function, one can use VLMs to lift features from 2D scene images, then associate it with another model to the 3D points. The final reward can be some function that encourages detections around features that correspond to VLM features. This would be the most direct analogy to how the work [1] utilized the VLM features. However, this would require a 2D-to-3D model, since all VLMs are currently trained under the 2D image domain, and at this moment, none exist for 3D data. In this way, our current reward formulation would be the most simple and straightforward, but we are incredibly encouraged if such a potential signal should, in the future, become available. We thank the reviewer for pointing out this interesting work and will expand on discussion regarding it. [1] Zhao, Shiyu, et al. "Exploiting unlabeled data with vision and language models for object detection." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022. [2] https://www.dimensions.com/classifications/transport [3] Fryar CD, Kruszon-Moran D, Gu Q, Ogden CL. Mean Body Weight, Height, Waist Circumference, and Body Mass Index Among Adults: United States, 1999-2000 Through 2015-2016. Natl Health Stat Report. 2018 Dec;(122):1-16. PMID: 30707668.
Decision
Accept (poster)