End-to-End Autonomous Driving without Costly Modularization and 3D Manual Annotation

Summary

This paper handles the costly modularization and 3D manual annotation in current end-to-end autonomous driving, which proposes an unsupervised pretext task to provide necessary environmental information, as well as a direction-aware training strategy to enhance the robustness in safety-critical steering scenarios. The authors conduct comprehensive experiments in both open- and closed-loop evaluation benchmarks, which demonstrate the effectiveness in various metrics. Moreover, the improvements are obtained with much less resource cost and faster inference speed, which is surprising and impressive. In addition, this paper gives in-depth discussion and performance comparison about the usage of ego status in the open-loop evaluation of nuScenes. The considerable improvement in the intersection rate with the road boundary, which is proposed in recent BEV-Planner, again proves the superiority of the designed pretext task.

Strengths

Overall, I am rather positive on this paper. In particular, I really like the motivation of this work that aims at finding a solution to relieve the heavy annotation and computation overload in current end-to-end autonomous driving. I believe this paper can inspire other works and facilitate this field. The strengths in this work include: (1) Enough novelty. This paper introduces an innovative unsupervised pretext task to perceive the environment, which is completely different from other works that accumulate subtasks requiring massive 3D annotation and computation resources. (2) Good performance. This paper demonstrates excellent performance and fast inference speed in both open- and closed-loop evaluation compared with other end-to-end methods. In specific on the challenging metric, i.e., intersection rate in BEV-Planner, the proposed approach surpasses other methods by a considerable margin. This clearly shows the effectiveness and advantages of the proposed method. (3) Insightful analysis. The authors provide extensive experiments and analysis for the proposed method. I appreciate this. The experimental analysis with various ablation studies allows a better understanding of each module. Notably, the authors observe the different computation ways of open-loop evaluation metrics between ST-P3 and UniAD and provide performance comparison with different settings, showing the comprehensiveness. (4) Good writing and organization. This paper is well-written and organized. Each section has a clear motivation. It’s easy to follow the ideas. I enjoy reading the paper. Overall, I believe this paper is significant to the autonomous driving community because it shows new insights and directions in designing simple but effective E2EAD framework with SOTA performance.

Weaknesses

(1) In this work, the 2D ROIs are crucial for the designed pretext task. I noticed that the authors adopt the open-set 2D detector GroundingDINO to generate the ROIs. Then the results and discussion of using other third-party detectors should be presented. (2) The proposed method is shown to be efficient with the unsupervised pretext task and self-supervised training strategy, which is nice. It is suggested the authors show the influence of the training data volume (e.g., 25% and 50%).

Questions

In the appendix, the authors show that the proposed method still achieves comparative performance even without backbone pretraining, while UniAD dramatically degrades without the pretrained weights of BEVFormer. What do you think are the reasons causing this?

Rating

9

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

It is suggested to provide discussion of limitations and broader impact in the revision.

Summary

This paper addresses the limitations of current end-to-end autonomous driving models that still rely on modular architectures with manually annotated 3D data. The authors propose an unsupervised pretext task that eliminates the need for manual 3D annotations by predicting angular-wise spatial objectness and temporal dynamics. This is achieved through an Angular Perception Pretext that models the driving scene without the need for manual annotation. A self-supervised training approach is introduced to enhance the robustness of planning in steering scenarios. This strategy learns the consistency of predicted trajectories under different augmented views. UAD demonstrates significant improvements in performance over existing methods like UniAD and VAD in both open-loop and closed-loop evaluations. It achieves these improvements with reduced training resources (44.3% of UniAD) and faster inference speed (3.4× faster than UniAD).

Strengths

1. The paper introduces UAD, an unsupervised autonomous driving framework that eliminates the need for costly 3D manual annotations, which is a significant departure from traditional modular approaches. 2. The Angular Perception Pretext is an innovative approach to spatial-temporal understanding without manual labeling, offering a new perspective on autonomous driving perception. 3. The experiments conducted are comprehensive, including both open-loop and closed-loop evaluations, which demonstrate the method's effectiveness across different scenarios. The paper provides a detailed comparison with state-of-the-art methods like UniAD and VAD, showcasing the improvements in performance metrics, which adds to the quality of the research.

Weaknesses

1. UAD treats an entire sector as occupied when only a part of it contains an object. This seems imprecise. This could potentially lead to less accurate spatial understanding of the environment. In autonomous driving, overly coarse representations might result in the vehicle making less accurate decisions, such as unnecessary braking or incorrect path planning. Have the authors tried some open world segmentation models for more accurate spatial information? 2. The paper draft does not provide explicit evidence or analysis on whether UAD can indeed benefit from training on a larger scale of data. The authors could conduct experiments with varying sizes of datasets to empirically evaluate how performance metrics change as more data becomes available. This could provide insights into the benefits of scaling up.

Questions

1. it is not explicitly stated whether UAD and UniAD use the exact same training and test split within the nuScenes dataset. 2. As UAD only uses basic obstacles for training. I wonder how will UAD reacts to traffic lights, lane lines, and policeman's gestures? 3. What is the BEV area designed in the experiment?

Rating

8

Confidence

5

Soundness

4

Presentation

3

Contribution

3

Limitations

In the current draft, UAD might be limited to basic obstacle detection and does not extend to the interpretation of traffic signals.

Summary

This paper aims to discard the requirement of 3D manual annotation in end-to-end autonomous driving by the proposed angular perception pretext task. Besides, this paper proposes a direction-aware learning strategy consisting of directional augmentation and directional consistency loss. Finally, the proposed method UAD achieves superior performance in both open-loop and closed-loop evaluation compared with previous vision-based methods with much lower computation and annotation costs.

Strengths

1) This paper aims to discard the requirement of 3D manual annotation in end-to-end autonomous driving, which is important and meaningful for training larger end-to-end autonomous driving models at scale. I totally agree and appreciate this. 2) This paper proposes a direction-aware learning strategy, which will further improve prformance by self-supervised learning. 3) UAD is evaluated in both open-loop and closed-loop evaluation and different metrics (UniAD, VAD, and BEV-Planner).

Weaknesses

1) There is a lack of explanation on how to use ego status. Besides, there should be more experiments about the performance of UAD without ego status. 2) There is a lack of explanation on how many frames are fused and what method is used for temporal fusion (sliding window or streaming). 3) The angular perception pretext task introduces 2D detection information for perception learning on BEV features. According to Table 6, it seems that angular design is very important for UAD. However, BEV-Planner can achieve a not-so-bad result without any 3D manual annotation and 2D detection information. Therefore, verifying the effectiveness of angular design on BEV-Planner will be more convincing. 4) For the proposed angular design, I do not think is very novel. Because the effectiveness of the 2D detection auxiliary head has been verified in BEVFormerV2 [1] and StreamPETR [2], the UAD just converts 2D object detection to BEV segmentation. 5) For the proposed direction-aware learning strategy, although it is useful, it is a method of data augmentation in EBV space, which I do not think is very novel.

Questions

1) I have serious concerns about the use of ego status and temporal fusion. If UAD uses GT ego status when fusing many frames, because there is no cumulative error in the ego trajectory, it will lead to falsely high performance. A better solution is to use predicted ego status instead of GT ego status (such as SparseDrive [3]). 2) Why does the use of a 3D detection head lead to a decrease in performance? What will be the result if UAD uses the online mapping head? 3) What is the performance if UAD uses a 2D detection auxiliary head instead of an angular design? For example, angular queries can be obtained by 2D detection auxiliary head and depth prediction (such as Far3D [4]). [1] BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervision [2] Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection [3] SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representation [4] Far3D: Expanding the Horizon for Surround-view 3D Object Detection

Rating

4

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

See the weakness and questions.

Summary

The article proposes an end-to-end (E2EAD) autonomous driving method called UAD (Unsupervised Autonomous Driving), which achieves autonomous driving on a visual basis without the need for expensive modular design and 3D manual annotation. UAD aims to overcome the limitations of existing E2EAD models that mimic traditional driving stack module architectures. These models typically require carefully designed supervised perception and prediction subtasks to provide environmental information for planning, which require a large amount of high-quality 3D annotation data and consume significant computational resources during training and inference processes.

Strengths

1. The method is novel and a good direction for exploring the end-to-end model's dependence on 3D manual annotation. 2. The paper has rich ablation experiments to demonstrate the effectiveness of the method. 3. The paper has advantages in both speed and accuracy compared to previous articles

Weaknesses

1. The paper lacks sufficient comparison with other methods(such as Interfuser[1],DriveAdapter[2],DriveMLM[3],VADv2[4]), which acheve better closed-loop performance on carla town 05 long benchmark. 2. In Table 6, the angular design brings too much gain, especially in terms of collision rate (from 1.37% to 0.19% ). It's strange. The angular design is about how to encode the sensor data and should not have so much impact on collision rate. 3. Angular design is widely used in BEV-related works, like PolarFormer[5]. The paper lacks citation about these works.And it's not proper to regard it as the main contribution of the work. 4. The paper lacks a part to introduce the use of 2D tasks as auxiliary tasks. [1] Hao Shao, Letian Wang, Ruobing Chen, Hongsheng Li, and Yu Liu. Safety-enhanced autonomous driving using interpretable sensor fusion transformer. In Conference on Robot Learning, pages 726–737. PMLR, 2023 [2] Xiaosong Jia, Yulu Gao, Li Chen, Junchi Yan, Patrick Langechuan Liu, and Hongyang Li. Driveadapter: Breaking the coupling barrier of perception and planning in end-to-end autonomous driving. 2023 [3] Wenhai Wang, Jiangwei Xie, ChuanYang Hu, Haoming Zou, Jianan Fan, Wenwen Tong, Yang Wen, Silei Wu, Hanming Deng, Zhiqi Li, et al. Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving. arXiv preprint arXiv:2312.09245, 2023 [4] Shaoyu Chen, Bo Jiang, Hao Gao, Bencheng Liao, Qing Xu, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Vadv2: End-to-end vectorized autonomous driving via probabilistic planning. arXiv preprint arXiv:2402.13243, 2024 [5] Yanqin Jiang, Li Zhang, Zhenwei Miao, Xiatian Zhu, Jin Gao, Weiming Hu, Yu-Gang Jiang. PolarFormer: Multi-camera 3D Object Detection with Polar Transformer. AAAI 2023

Questions

Why use 2D boxes instead of 2D segmentation of objects? Is it more reasonable to use 2D segmentation labels to link the points in BEV space with the points in image space?

Rating

3

Confidence

5

Soundness

3

Presentation

4

Contribution

2

Limitations

1. Incompele comparison on CARLA benchmark. 2. Lack citation about angular design. 3. Angular design is not novel. 4. Some of the experiment results are not that convincing.

Summary

The paper proposes a new e2e driving model named UAD. In this paper, the authors propose an unsupervised method for effective training and inference of the e2e model. The paper mainly has two contributions: 1. It designs an angular-wise perception module. In this module, the authors directly project 2D GT labels onto the BEV and define a new BEV map label for perception training. This module design can efficiently reduce complexity and preserve effectiveness. 2. The authors propose a direction-aware method to augment the trajectory training and use consistency loss for further supervision. The final results show the effectiveness and soundness of the proposed method.

Strengths

1. The idea of the angular-wise perception module is interesting. It can utilize a huge amount of 2D annotated autonomous driving datasets to train the e2e model, which removes the restriction of the limited number of 3D annotations. 2. The proposed direction-aware method for trajectory prediction is also meaningful since it can add additional consistency loss for better supervision. 3. The results are promising and the efficiency improvement is impressive.

Weaknesses

1. The design of the angular-wise perception module is a little bit counter-intuitive to me. From my perspective, it works because (1) It can greatly enlarge the size of the training data. (2) It makes the perception task simpler, thus the model can do it better (knowing an object in a direction is much simpler than detecting the BBox). (3) The efficiency improves because of the light design of the perception task. I think it can be simply treated as a low-resolution object detection task without depth. Except for (1), I do not understand why it can improve the final results. 2. The design of the direction-aware planning training strategy is effective but simple. I cannot see too much insight here. Could the authors provide more insight into this? Or is this just an engineering trick? 3. For experiments, do you use exactly the same training data for both the open-loop and close-loop experiments? If yes, could you provide more analysis about why the results are surprisingly good even if you use a low-resolution detection module? If not, could you provide details about the pertaining data info? Can the impressive results come from leaked data? How to prevent testing data leakage? 4. In real-world applications, when the vehicle plans its path, it needs to "see" a lot of things, including objects and some other elements, like traffic lights, traffic signals, or some special marks on the road. How can you handle these elements based on your model? For example, can your model understand traffic signals and see red traffic lights? If not, how to extend your model to real-world scenarios and applications?

Questions

Please see the weakness part for questions and suggestions.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

My main concern with this paper mainly comes from the insight of the angular-wise perception module. I still cannot understand why it works except for the huge amount of additional training data. Provide more details for this. Besides, how to deploy and extend the model to real-world cases, that requires depth information (e.g., ) or semantic information (e.g., traffic light)? The authors should provide more discussion about this to make the work more promising.

Help Check the Rebuttal, Make Discussions, and Update the Final Recommendations

Dear Reviewers, Thanks for serving as a reviewer for the NeurIPS. The rebuttal deadline has just passed. The author has provided the rebuttal, could you help check the rebuttal and other fellow reviewers' comments, make necessary discussions, and update your final recommendations as soon as possible? Thank you very much. Best, Area Chair

Final comments

Thank the authors for the detailed response and information. Some of my concerns are addressed, but I still have problems with the design and working principles of the angular-wise perception module. I have strong concerns about its actual application values in real-world scenes. I will slightly raise the score, and the authors are encouraged to further refine the paper and make it more solid.

Post Rebuttal Comments

Thanks the authors for their detailed feedback. All my concerns have been addressed. I very like this paper due to its novelty, great performance and simplicity. I believe this is a potentially right way for end-to-end autonomous driving. I also see other reviewers' comments and the authors' rebuttals, and think that the authors have done a good job to explain their work better. Thus, I keep my original score (Very Strong Accept).