Summary
The paper proposes ActiveAD, an active learning framework for end-to-end autonomous driving. One major challenge in E2E-AD lies in the expensive data annotation process and the long-tailed distribution of AD datasets, where much of the collected data is redundant (e.g., straightforward driving on empty roads). ActiveAD addresses these issues by designing the following metrics. Ego-Diversity: A diversity-based initialization method that considers weather, lighting, and driving behaviors to address cold-start issues. Planning-Oriented Metrics: The use of Displacement Error, Soft Collision, and Agent Uncertainty metrics for iterative sample selection to reduce the annotation burden while maintaining high planning performance. The paper demonstrates data-efficiency improvements by achieving state-of-the-art performance using only 30% of training data on both the nuScenes dataset and CARLA simulation.
Strengths
1. This paper studies the problem of active learning in self-driving. It is an important problem for scalable developments and faster iterations for industry. The focus on planning performance for active learning is new. Existing active learning methods mainly optimize perception / prediction tasks, but ActiveAD extends this to planning in E2E-AD.
2. The paper is well written and easy to follow. The metrics proposed in this paper are intuitive and straightforward. The combination of displacement error, soft collision, and agent uncertainty provides a robust way to identify critical data samples for annotation. I also like thorough ablation studies presented in the paper.
Weaknesses
1. Insufficient evaluation and limited generalization to real-world scenarios. It is particularly important for this paper to demonstrate the proposed metrics can be adapted to different datasets and architectures rather than just some heuristics-based tuning on specific datasets. It is not a big surprise that using 30% data can achieve on-par performance with careful data selection. Moreover, this paper does not evaluate the robustness of the trained autonomy on more extreme / out-of-distribution (OOD) settings (e.g., safety-critical scenarios, extreme weather, complex interactions, etc). I do not believe the metrics on the eval set can tell the full story. In general, my biggest concern is how generic the proposed metrics are and how robust the trained model is. Current evaluation on nuScenes and CARLA is not sufficient. I would recommend testing on larger datasets (e.g., argoverse, waymo) and more diverse e2e models.
2. This paper misses a significant amount of work for active learning in the self-driving domain. For instance, the seminal work is not discussed in the paper. There are also a lot of follow up works (e.g., [2][3]). More comparisons and discussions with them would be beneficial. Another baseline to consider is getting some planning costs for each scene and pick the hardest ones.
[1] Scalable Active Learning for Object Detection. Haussmann et al., 2020. \
[2] Just Label What You Need: Fine-Grained Active Selection for Perception and Prediction through Partially Labeled Scenes. Segal et al., 2021. \
[3] Improving the Intra-class Long-Tail in 3D Detection via Rare Example Mining. Jiang et al., 2022.
3. There are too many parameters to tune and the procedure is quite complicated. How can we make sure we choose the correct setting when in the production setup (say we need to train a new model based on newly collected data and we cannot tune). I am a bit worried about the real impact of the proposed paradigm as there is no automatic data selection procedure involved like many other works (e.g., using the training loss, entropy in the prediction etc). Also, it seems that the planning improvement is quite limited when training more data (30% -> more data) but perception and prediction can continue improve from Table 6. The mAP with 30% data is only 15.85 vs 26.65 which is a signficant performance drop. I am worried that using this paradigm will give us less robust models (the planning metrics can be noisy and cannot tell the full story?).
4. There are a lot of places where the bold highlights are wrong. For instance, in Table 3, on night scenario, coreset results 0.97 / 0.27 is actually better than the ActiveAD. In rainy and turn right, coreset results 0.06, 0.78 are better ActiveAD. In Table 1, VAD-Tiny 20% data, VAAL average collision error for 1s is smaller. I recommend the authors to carefully check the tables.