Navigation-Guided Sparse Scene Representation for End-to-End Autonomous Driving

Thanks and Response to Reviewer 4p2A (1/2)

Thanks for your valuable review which helps us improve this work. We address your concerns below. ### **Closed-Loop Experiments** We appreciate the reviewer's emphasis on closed-loop evaluation. We conduct closed-loop experiments using the CARLA simulator \[1\], leveraging the widely adopted Town05 Long benchmark to evaluate performance. The training dataset consists of 189K frames collected by Roach \[2\] at 2 Hz across 4 CARLA towns (Town01, Town03, Town04, and Town06), following previous works \[3-5\]. The training data has no overlap with Town05 Long benchmark. We utilized the official CARLA metrics for evaluation: - Route Completion (RC): Percentage of the route completed. - Infraction Score (IS): Measures infractions, including collisions with pedestrians, vehicles, road layout, and traffic signals. - Driving Score (DS): Main metric, calculated as the product of RC and IS. We utilize ResNet-34 as the image backbone, resizing the input image size to 900 $\times$ 256. For STL module, we concatenate the command, target point and current speed to a MLP as navigation information. The TCP head \[3\] is applied for planning module. Below are our closed-loop results, which will be added to the revised paper detailedly. **We also attach the video clips in supplementary material.** | Method | Modality | DS $\uparrow$ | RC $\uparrow$ | IS $\uparrow$ | | --- | :---: | :---: | :---: | :---: | | CILRS | Camera | 7.8 | 10.3 | 0.75 | | LBC | Camera | 12.3 | 31.9 | 0.66 | | Transfuser | Camera&LiDAR | 31.0 | 47.5 | 0.77 | | Roach | Camera | 41.6 | **96.4** | 0.43 | | ST-P3 | Camera | 11.5 | 83.2 | \- | | LAV | Camera&LiDAR | 46.5 | 69.8 | 0.73 | | TCP | Camera | 57.2 | 80.4 | 0.73 | | VAD | Camera | 30.3 | 75.2 | \- | | ThinkTwice | Camera&LiDAR | 65.0 | 95.5 | 0.69 | | DriveAdapter | Camera&LiDAR | 65.9 | 94.4 | 0.72 | | **SSR** | Camera | **78.9** | 95.5 | **0.83** | Our method significantly outperforms existing works in terms of DS, including those utilizing LiDAR input \[4, 5\]. For camera-based methods, SSR achieves a 31.7-point improvement in DS over TCP and a remarkable 2.6$\times$ increase over VAD. The closed-loop experiments demonstrate that our method performs reliably in complex, long-term driving scenarios, reinforcing the viability of eliminating middle supervision without compromising on safety or accuracy. To further investigate the impact of our proposed modules, we perform an ablation study in the closed-loop setting. The results below demonstrate that both modules contribute significantly to performance. The STL module yields substantial improvements across all metrics while BWM module further enhances the driving score and infraction score. | STL | BWM | DS $\uparrow$ | RC $\uparrow$ | IS $\uparrow$ | | --- | --- | :---: | :---: | :---: | | | | 62.6 | 90.3 | 0.72 | | ✔ | | 72.7 | **97.4** | 0.75 | | ✔ | ✔ | **78.9** | 95.5 | **0.83** | ### **Comparison with BEV-Planner** We believe the comparison with BEV-Planner \[6\] was ignored in the initial review. In fact, we **did include BEV-Planner results in Tab. 1** (Line 343). The comparison of Tab. 1 is based on two different protocols: - The upper part compares methods using the UniAD protocol. - The lower part compares methods using the VAD protocol. This distinction is explained in Lines 314-321. BEV-Planner appears only in the lower section because it reports results using the VAD protocol in its paper. Due to the lack of released code, we could not include UniAD protocol results for BEV-Planner. We ensured a fair and consistent comparison by maintaining protocol alignment within each section of the table. We hope this clarifies the situation. The omission of BEV-Planner from the upper part of the table is due to protocol differences and not an oversight on our part.

Thanks and Response to Reviewer 4p2A (2/2)

### **Evaluation on nuScenes** We noted the significant impact of using ego status in the planning module on L2/CR metrics revealed in \[6\], so we did not incorporate it in our experiments to ensure a robust design. Additionally, BEV-Planner introduced two solutions for more adequate nuScenes evaluation: - The introduction of Curb Collision Rate (CCR) metric, which methods relying on ego status may struggle. - The distinction between go-straight (GS) and turn left/right (LR) scenarios when calculating L2 and CR metrics, which provides a finer-grained assessment. These solutions are both applied in our experiments in Tab.3 and Tab. 4. In response to the reviewer’s concerns on the open-loop nuScenes results, we provide more CCR comparisons and scenario-based results in VAD protocol below: | Method/CCR (%) | 1s $\downarrow$ | 2s $\downarrow$ | 3s $\downarrow$ | Avg $\downarrow$ | | --- | :---: | :---: | :---: | :---: | | UniAD | 0.21 | 1.32 | 3.63 | 1.72 | | VAD-Base | 0.60 | 2.38 | 5.18 | 2.72 | | GoStraight | 2.07 | 8.09 | 15.7 | 8.62 | | Ego-MLP | 0.27 | 2.52 | 6.60 | 2.93 | | BEV-Planner | 0.70 | 3.77 | 8.15 | 4.21 | | **SSR** | **0.15** | **0.46** | **1.31** | **0.64** | | Method | Avg L2-GS (m) $\downarrow$ | Avg L2-LR (m) $\downarrow$ | Avg CR-GS (%) $\downarrow$ | Avg CR-LR (%) $\downarrow$ | | --- | :---: | :---: | :---: | :---: | | BEV-Planner | 0.48 | 0.81 | 0.40 | 2.25 | | SSR | **0.37** | **0.53** | **0.04** | **0.19** | These comparisons demonstrate that SSR is effective in both straightforward and complex scenarios on nuScenes, supporting its adaptability beyond simple cases. We appreciate the opportunity to discuss the role of open-loop evaluation. While we recognize the limitations of open-loop testing, we believe that dismissing it entirely overlooks valuable insights. The crux lies in ensuring an appropriate distribution of testing data and relevant metrics, as also suggested by recent research in simulator design, such as NAVSIM \[7\]. In this spirit, we believe our results on the nuScenes dataset, alongside our visualizations, provide meaningful insights into our model’s design and functionality. \[1\] CARLA: an open urban driving simulator. CoRL, 2017 \[2\] End-to-end urban driving by imitating a reinforcement learning coach. ICCV, 2021 \[3\] Trajectory-guided control prediction for end-to-end autonomous driving: a simple yet strong baseline. NeurIPS, 2022 \[4\] Think twice before driving: towards scalable decoders for end-to-end autonomous driving. CVPR, 2023 \[5\] Driveadapter: Breaking the coupling barrier of perception and planning in end-to-end autonomous driving. ICCV, 2023 \[6\] Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving? CVPR 2024 \[7\] NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking. NeurIPS, 2024

Thanks and Response to Reviewer zc5J (1/2)

Thank you for your helpful comments. Following are our detailed responses to your concerns. ### **Clarification of Navigation Information** As the reviewer observed, navigation information is often overlooked in prior academic work, and to our knowledge, its use in industry is also primarily limited to the planning module. In the contrast, our method **leverages navigation information to directly guide the perception process**, rather than just serving as input for the planning module. This distinction is visually illustrated in Fig. 2 of our paper. We understand the reviewer’s observation that improvements in lane-following scenarios dominate the ablation results in Table 2. This is partially due to dataset bias in nuScenes, where lane-keeping scenarios are prevalent. To address this, we additionally conduct closed-loop evaluations using the CARLA simulator on the widely adopted Town05 Long benchmark. For evaluation, we utilized official CARLA metrics: - Route Completion (RC): Percentage of the route completed. - Infraction Score (IS): Measures infractions, including collisions with pedestrians, vehicles, road layout, and traffic signals. - Driving Score (DS): Main metric, calculated as the product of RC and IS. The results below demonstrate that both modules contribute significantly to performance. The STL module significantly improves all metrics, while the BWM module further enhances DS and IS: | STL | BWM | **DS** $\uparrow$ | RC $\uparrow$ | IS $\uparrow$ | | --- | --- | :---: | :---: | :---: | | | | 62.6 | 90.3 | 0.72 | | ✔ | | 72.7 | **97.4** | 0.75 | | ✔ | ✔ | **78.9** | 95.5 | **0.83** | Furthermore, we want to clarify that the effect of navigation guidance is evaluated in Tab. 3 instead of Tab.2 of our paper. The experiments in Tab. 3 demonstrate that navigation guidance improves planning results across all scenarios, rather than only lane following scenarios. ### **Clarification of BWM Module** Despite the advances of STL module, our navigation-guided perception design introduces unique challenges in temporal modeling. As noted in Lines 284-286, scene queries may focus on different elements in consecutive frames. Prior query-level pairwise temporal modeling approaches \[1, 2\] fail to address this mismatch, causing non-convergence in our architecture. To overcome this limitation, our novel BWM module leverages BEV-level self-supervision to stabilize features across frames, achieving a further improvement in collision rate (Tab. 2) and outperforming prior methods (Tab. 1). Additionally, the component-wise ablation in closed-loop experiments emphasizes the importance of the BWM module, as shown in the table above. We understand the reviewer’s concern regarding the term “world model.” Our "world model" is applied to BEV features across consecutive frames, which compensates the absence of perception task supervision. We recognize that this term might appear too broad, as the module is primarily focused on predicting the next frame in BEV rather than building a full world model. If this naming is confusing, we are open to renaming it to something more descriptive, such as **"BEV Prediction Module"** or **"Future Feature Predictor"**, to better reflect its function in predicting the future state of the environment.

Thanks and Response to Reviewer zc5J (2/2)

### **Contribution and Novelty of SSR** The design of the STL and BWM modules is grounded in leveraging the **planning-focused nature of E2E frameworks.** Specifically, inspired by human drivers' attention mechanisms, we propose that E2EAD systems can selectively focus on relevant elements, rather than extracting all scene elements exhaustively in previous works. The primary contribution of our paper lies in establishing a human-inspired framework for E2EAD, significantly reduces computational costs by adaptively focusing on essential parts of scenes. This shift on paradigm emphasizes planning-centric design over exhaustive perception, presenting an more efficient and practical alternative to traditional architectures. Within this framework, the STL module implements the selective perception strategy, while the BWM module enhances temporal coherence across frames. These two modules are integral and complementary components of our novel navigation-guided perception design of SSR. Their contributions should therefore be assessed in the context of the overall framework rather than in isolation. We also provide the results in closed-loop experiments below, while attaching the video clips in supplementary material. | Method | DS $\uparrow$ | RC $\uparrow$ | IS $\uparrow$ | | --- | :---: | :---: | :---: | | CILRS | 7.8 | 10.3 | 0.75 | | LBC | 12.3 | 31.9 | 0.66 | | Transfuser | 31.0 | 47.5 | 0.77 | | Roach | 41.6 | **96.4** | 0.43 | | ST-P3 | 11.5 | 83.2 | \- | | LAV | 46.5 | 69.8 | 0.73 | | TCP | 57.2 | 80.4 | 0.73 | | VAD | 30.3 | 75.2 | \- | | ThinkTwice | 65.0 | 95.5 | 0.69 | | DriveAdapter | 65.9 | 94.4 | 0.72 | | **SSR** | **78.9** | 95.5 | **0.83** | The superior performance of SSR in both open-loop and closed-loop experiments demonstrates its capability to explore deeper into the potential of E2EAD systems. This novel framework, which integrates navigation-guided selective perception and temporal coherence, goes beyond the simple introduction of navigation information. It sets a new benchmark for practical and efficient autonomous driving solutions. ### **Color Scheme** Our figures utilize a white background for traditional elements and a colored background to highlight the unique contributions of our modules. We appreciate the reviewer’s suggestion regarding aesthetics and will incorporate improvements in the revised version to enhance clarity and appeal. For instance: - In Fig. 2, the white elements will be replaced with a more visually distinct background to emphasize critical features. - In Fig. 4, module colors will be adjusted to correspond to their respective blocks, using a cohesive color scheme (e.g., red hues for STL and blue hues for BWM). \[1\] Enhancing end-to-end autonomous driving with latent world model. arXiv \[2\] End-to-end autonomous driving without costly modularization and 3d manual annotation. arXiv

Thanks and Response to Reviewer rfNp

Thank you for your thoughtful comments and constructive feedback. We address each of your points below: **Response to W1 & Q1:** Our model builds upon the VAD-Tiny [1] architecture. Specifically, we utilize ResNet-50 as the image backbone,  while the BEV encoder adopts the approach from BEVFormer [2]. To enhance navigation awareness, we modify the TokenLearner [3] into BEV space to extract scene queries. The planning decoder incorporates cross-attention, while the BWM module predominantly employs self-attention. We will release the code for our full model structure, providing further clarity on the implementation. In our experiments on the nuScenes dataset, we use three navigation commands (go straight, turn left, turn right). The preprocessing of these commands follows prior practice in VAD by transforming ground truth ego trajectories into driving commands based on the final future trajectory offset. This approach is also documented in the open-source VAD [code](https://github.com/hustvl/VAD/blob/main/tools/data_converter/vad_nuscenes_converter.py#L452). ```python # drive command according to final fut step offset from last context frame if ego_fut_trajs[-1][0] >= 2: command = np.array([1, 0, 0]) # Turn Right elif ego_fut_trajs[-1][0] <= -2: command = np.array([0, 1, 0]) # Turn Left else: command = np.array([0, 0, 1]) # Go Straight ``` **Response to W2:** We appreciate your suggestion to report failure cases. Our appendix (Line 749, Page 14) has provided visualizations and detailed analysis of two common failure types, one of which is related to ambiguous navigation commands. **Response to Q2:** Since end-to-end autonomous driving (E2EAD) models typically lack HD maps, a high-level driving command input is necessary for navigation. To address the reviewer's concerns about potentially confusing commands, we conducted additional experiments alongside our failure case analysis. | Command | Avg L2 (m) $\downarrow$ | Avg CR (%) $\downarrow$ | | --- | --- | --- | | Go Straight | 0.77 | 0.23 | | Turn Left | 0.78 | 0.22 | | Turn Right | 0.8 | 0.53 | | Random | 0.78 | 0.31 | | Original | **0.75** | **0.15** | For these experiments, we tested four additional command types: all go straight, all turn left, all turn right, and random commands. Notably, as the ego vehicle typically operates on the left side of the road, "turn left" and "go straight" commands led to comparable L2 error and collision rates. However, the "turn right" command showed an obvious increase in collision rate, often resulting in conflicts with oncoming vehicles. Random commands caused a noticeable degradation in performance but still produced reasonable results, demonstrating the model’s resilience to noisy navigation inputs. These findings highlight the strengths of our approach while identifying opportunities to improve its handling of ambiguous or conflicting commands. [1] Vad: Vectorized scene representation for efficient autonomous driving. In ICCV, 2023. [2] Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers. In ECCV, 2022 [3] Tokenlearner: Adaptive space-time tokenization for videos. In NeurIPS, 2021

Thanks and Response to Reviewer pPL3

Thank you for your valuable comments and constructive feedback. We address each of your points below: **Response to W1:** We appreciate your feedback regarding the term "world model" for the BWM module. We recognize that this term might appear too broad, as the module is primarily focused on predicting the next frame in BEV (Bird's Eye View) rather than building a full world model. If this naming is confusing, we are open to renaming it to something more descriptive, such as **"BEV Prediction Module"** or ****"Future Feature Predictor"****, to better reflect its function in predicting the future state of the environment. **Response to W2:** Thank you for suggesting theoretical analysis. Our work is primarily empirical; however, we aim to make SSR’s navigation-guided perception process more interpretable by visualizing scene queries. These visualizations reveal how SSR’s perception module prioritizes scene elements based on navigational input, distinguishing it from traditional BEV architectures. **Response to Q1:** We note the significant impact of using ego status in the planning module on L2/CR metrics revealed in \[1\] and \[2\], so we did not incorporate it in our experiments to ensure a robust design. Regarding the reviewer's concern about the nuScenes dataset, we conducted additional closed-loop experiments using the CARLA simulator, leveraging the widely adopted Town05 Long benchmark to evaluate performance. The training dataset consists of 189K frames collected by Roach \[3\] at 2 Hz across 4 CARLA towns (Town01, Town03, Town04, and Town06), following previous works \[4-6\]. The training data has no overlap with Town05 Long benchmark. We utilized the official CARLA metrics for evaluation: - Route Completion (RC): Percentage of the route completed. - Infraction Score (IS): Measures infractions, including collisions with pedestrians, vehicles, road layout, and traffic signals. - Driving Score (DS): Main metric, calculated as the product of RC and IS. We utilize ResNet-34 as the image backbone, resizing the input image size to 900 $\times$ 256. For STL module, we concatenate the command, target point and current speed to a MLP as navigation information. The TCP head \[4\] is applied for planning module. Below are our closed-loop results, which will be added to the revised paper. **We also attach the video clips in supplementary material.** | Method | Modality | DS $\uparrow$ | RC $\uparrow$ | IS $\uparrow$ | | --- | :---: | :---: | :---: | :---: | | CILRS | Camera | 7.8 | 10.3 | 0.75 | | LBC | Camera | 12.3 | 31.9 | 0.66 | | Transfuser | Camera&LiDAR | 31.0 | 47.5 | 0.77 | | Roach | Camera | 41.6 | **96.4** | 0.43 | | ST-P3 | Camera | 11.5 | 83.2 | \- | | LAV | Camera&LiDAR | 46.5 | 69.8 | 0.73 | | TCP | Camera | 57.2 | 80.4 | 0.73 | | VAD | Camera | 30.3 | 75.2 | \- | | ThinkTwice | Camera&LiDAR | 65.0 | 95.5 | 0.69 | | DriveAdapter | Camera&LiDAR | 65.9 | 94.4 | 0.72 | | **SSR** | Camera | **78.9** | 95.5 | **0.83** | Our method significantly outperforms existing works in terms of DS, including those utilizing LiDAR input \[5, 6\]. For camera-based methods, SSR achieves a 31.7-point improvement in DS over TCP and a remarkable 2.6$\times$ increase over VAD. These results indicate the comprehensive capabilities of SSR in long-term driving scenarios. We also appreciate your interest in extending SSR to more complex navigation commands, as noted in our limitations. We plan to explore such extensions in future work, potentially incorporating large language models (LLMs) to enable natural language-based navigation tasks. **Response to Q2:** The visualizations in Figure 5 are actually in square grids, not hexagonal format. The hexagonal appearance arises from the lower attention values in the grid corners, which may appear hexagonal due to the natural gradient in the attention map. We apologize for any confusion and will clarify this in our figure caption in revised version. For more comprehensive visualizations, please refer to Figure 10 (page 13) in the appendix, where we provide the full attention map set. \[1\] Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes. arXiv \[2\] Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving? CVPR 2024 \[3\] End-to-end urban driving by imitating a reinforcement learning coach. In ICCV, 2021 \[4\] Trajectory-guided control prediction for end-to-end autonomous driving: a simple yet strong baseline. NeurIPS, 2022 \[5\] Think twice before driving: towards scalable decoders for end-to-end autonomous driving. In CVPR, 2023 \[6\] Driveadapter: Breaking the coupling barrier of perception and planning in end-to-end autonomous driving. In ICCV, 2023

Thanks and Response to Reviewer odD2

Thanks for your helpful comments. We address your concern below: **Response to W1 & Q2:** We appreciate the reviewer’s concerns regarding the number of scene queries. A key strength of our method lies in demonstrating that a minimal set of scene queries can effectively represent the entire scene without compromising performance. Our visualizations provide insight into how such a small number of queries can focus on critical traffic units and salient regions. Even in extremely dense scenes, we believe the number of interactive traffic units (e.g., key agents, critical road features) remains relatively small compared to the total scene complexity. This allows the model to handle dense environments without requiring a large number of queries. The results in Table 2(b) further support this, showing that increasing the number of scene queries beyond a certain point leads to suboptimal performance. However, we agree with the reviewer that adopting a dynamic number of scene queries could further enhance the model's flexibility and performance. A possible direction would be to start with a larger initial set of queries and predict a confidence score for each one. Queries falling below a predefined threshold could be discarded, enabling an adaptive number of queries tailored to the scene’s complexity. We leave this promising direction for further exploration. **Response to W2&Q3:** We appreciate the reviewer’s observation regarding the absence of perception metrics in Table 4. To address this, we provide the detailed results below: | Map | Obs | $mAP_{Map} \uparrow$ | $mAP_{Obs} \uparrow$ | $NDS_{Obs} \uparrow$ | | --- | --- | :---: | :---: | :---: | | | | \- | \- | \- | | ✔ | | 10.13 | \- | \- | | | ✔ | \- | 4.18 | 14.4 | | ✔ | ✔ | 10.72 | 3.96 | 12.67 | For fairness, we followed the 12-epoch training setup described in Line 360 of our revised paper. While this ensures consistency across experiments, it leads to lower perception performance compared to previous methods that train perception modules for 48 epochs before planning module's learning. To further address this concern, we conducted additional experiments by training SSR with pretrained weight from 48 epochs perception learning, as used in VAD. The pretrained model achieved 27.99 mAP and 40.15 NDS for obstacles and 48.78 mAP for mapping on nuScenes. However, even with these pretrained weights, SSR did not show significant improvements in trajectory prediction. This suggests that SSR inherently **learns scene understanding in a different way** than traditional perception-tasks-driven approaches. | Pretrain | Avg L2 (m) $\downarrow$ | Avg CR (%) $\downarrow$ | Avg CCR (%) $\downarrow$ | | --- | --- | :---: | :---: | | ✔ | 1.11 | 0.66 | 2.10 | | $\times$ | 0.75 | 0.15 | 1.30 | These results reinforce SSR’s distinct capability to bypass traditional perception modules, emphasizing its end-to-end nature and efficiency. **Response to W3:** To address the reviewer's concerns on nuScenes dataset, we additionally conducted closed-loop evaluations in the CARLA simulator, which demonstrate SSR’s robustness in diverse settings. We utilized the official CARLA metrics for evaluation: - Route Completion (RC): Percentage of the route completed. - Infraction Score (IS): Measures infractions, including collisions with pedestrians, vehicles, road layout, and traffic signals. - Driving Score (DS): Main metric, calculated as the product of RC and IS. Below results are evaluated on widely adopted Town05 Long benchmark. **We also attach the video clips in supplement materials.** | Method | Modality | DS $\uparrow$ | RC $\uparrow$ | IS $\uparrow$ | | --- | :---: | :---: | :---: | :---: | | CILRS | Camera | 7.8 | 10.3 | 0.75 | | LBC | Camera | 12.3 | 31.9 | 0.66 | | Transfuser | Camera&LiDAR | 31.0 | 47.5 | 0.77 | | Roach | Camera | 41.6 | **96.4** | 0.43 | | ST-P3 | Camera | 11.5 | 83.2 | \- | | LAV | Camera&LiDAR | 46.5 | 69.8 | 0.73 | | TCP | Camera | 57.2 | 80.4 | 0.73 | | VAD | Camera | 30.3 | 75.2 | \- | | ThinkTwice | Camera&LiDAR | 65.0 | 95.5 | 0.69 | | DriveAdapter | Camera&LiDAR | 65.9 | 94.4 | 0.72 | | **SSR** | Camera | **78.9** | 95.5 | **0.83** | The superior performance of SSR in both open-loop and closed-loop experiments demonstrates its capability to explore deeper into the potential of E2EAD systems. **Response to W4:** We appreciate the reviewer’s suggestion and add more detailed explanation of the colormap in Line 466 of revised paper. The colormap represents the **navigation-aware BEV features**, where warmer colors indicate higher attention weights. **Response to Q1:** The channel dimension of each scene query is **256**, following standard practice in previous works. We have added its dimension clearly in our revised paper.