Summary
This paper presents FusionAD, a hierarchical multi-task autonomous driving network in the style of recent work UniAD for tracking, mapping, motion prediction, occupancy and open-loop planning. The UniAD model is extended in 3 ways: (1) an updated backbone network which is based on FusionFormer (camera + LiDAR + time) instead of BEVFormer (camera + time); (2) a modified prediction module with a multi-stage predict-and-refine formulation; and (3) a modified planning module with a new collision loss during training and ego status input. Besides this, the architecture is trained in a new 3-stage process. This leads to an approach with strong empirical performance across all tasks on the nuScenes dataset.
Strengths
The key strengths of the paper are the strong empirical results of the final model and standardized experimental setup with established data and metrics, significantly pushing the state-of-the-art. The results are well-structured, demonstrating improvements both quantitatively and qualitatively. Besides this, the proposed modifications are simple and well-motivated and the paper is clearly written.
Weaknesses
1. Inconsistent depiction of novelty over UniAD in Fig. 1b: UniAD has camera-only features, but an identical hierarchical structure for perception, prediction and planning as the proposed FusionAD from Fig. 1c. The current figure creates the impression of a new contribution with respect to the downstream architecture, while in the paper the only contributions are towards existing modules in the UniAD framework (backbone, forecasting, planning).
2. Table 6 shows that gains in open-loop planning performance over UniAD result from only the new ego-status input. Recent work has already shown that including ego-status as a planner input significantly simplifies the task [1,2], leading to SoTA results on nuScenes even with ego-status as the only input [1]. The discussion should clearly mention that the new sensor fusion backbone does not contribute to open-loop planning performance.
3. Unclear significance of proposed changes to MotionFormer for prediction: Table 5 shows a very minor impact towards motion forecasting performance with the proposed changes (0.394→0.388 MinADE), which means that nearly all gains in performance with respect to UniAD (0.708→0.388 MinADE) come from the FusionFormer based backbone.
4. Missing related work: the idea of multi-modal fusion in multi-task end-to-end autonomous driving stacks is well-established in all the top entries of the CARLA simulator [3,4,5,6,7] but these are not discussed in the paper. In Section 2.3, it is incorrectly claimed that TransFuser does not use BEV space, even though the BEV branch is a key component of the TransFuser architecture.
[1] https://arxiv.org/abs/2303.12077
[2] https://arxiv.org/abs/2305.10430
[3] https://arxiv.org/abs/2203.11934
[4] https://arxiv.org/abs/2205.15997
[5] https://arxiv.org/abs/2207.14024
[6] https://arxiv.org/abs/2305.10507
[7] https://arxiv.org/abs/2306.07957
Questions
1. Please see “Weaknesses” - these are the key points with the most influence on my rating. Taken together, W2 and W3 show that the proposed changes to the prediction and planning modules provide limited insights.
2. Unclear separation of methods in Table 4: different methods in Table 4 have different input modalities (camera, LiDAR, ego status). Please add columns clearly showing this difference.
3. Would it be possible to provide a comparison to UniAD in terms of model size, training time and inference time? This is valuable information from a practitioner’s perspective.
4. I would recommend making the differentiation between UniAD and FusionAD clearer in Fig. 3 and Fig. 4 by highlighting the new architectural components.
Rating
3: reject, not good enough
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.