Trajectory prediction plays a vital role in the performance of autonomous driving systems, and prediction accuracy, such as average displacement error (ADE) or final displacement error (FDE), is widely used as a performance metric. However, a significant disparity exists between the accuracy of predictors on fixed datasets and driving performance when the predictors are used downstream for vehicle control, because of a dynamics gap. In the real world, the prediction algorithm influences the behavior of the ego vehicle, which, in turn, influences the behaviors of other vehicles nearby. This interaction results in predictor-specific dynamics that directly impacts prediction results. In fixed datasets, since other vehicles' responses are predetermined, this interaction effect is lost, leading to a significant dynamics gap. This paper studies the overlooked significance of this dynamics gap. We also examine several other factors contributing to the disparity between prediction performance and driving performance. The findings highlight the trade-off between the predictor's computational efficiency and prediction accuracy in determining real-world driving performance. In summary, an interactive, task-driven evaluation protocol for trajectory prediction is crucial to capture its effectiveness for autonomous driving. Source code along with experimental settings is available online.
Paper
References (40)
Scroll for more · 28 remaining
Similar papers
Peer review
Summary
This paper presents a timely and well-executed study on motion prediction, focusing on the relationship between the dominant evaluation paradigm (static, offline metrics) and the true objective of research in this domain (safer planning). The study uncovers several important and surprising findings: (1) static offline metrics are not correlated to planning performance, (2) evaluating on the frames observed by the planner (dynamic offline metrics) are significantly better correlated, (3) computational efficiency in prediction is increasingly important for sophisticated planners, and (4) simple baselines for prediction (constant velocity/constant acceleration) perform best when considering downstream planning performance.
Strengths
The study setup is excellent, making well-motivated choices for the task, simulator, predictors, planners, metrics, etc. The writing is clear. The presentation of the key findings both visually and numerically also makes this paper an interesting and easy-to-understand read. Most importantly, it highlights a flaw in the widely used metrics that the community optimizes motion prediction models for, and shows that naive baselines are sufficient to outperform SoTA learned forecasting when using the right metrics.
Weaknesses
Overall, the paper has very few weaknesses in my opinion. While the selected ML prediction models do not include the current leaderboard winners, I believe they are representative. The experiments could have been conducted on more realistic scenarios (e.g. the nuPlan simulator), but Section 5.1 shows that the simulator used is sufficiently aligned to real datasets for the purpose of this study. However, some technical details regarding the study are unclear (please see the “Questions” section #2, #3, #4 for specifics).
Questions
1. A very interesting finding that I suggest should be highlighted more is that in all closed-loop evaluation settings, the best among the prediction methods used in this study is always either CV or CA - the naive baselines. 2. Do the CV and CA baselines forecast the positions along a straight line (heading direction), or do they take into account angular speed/acceleration as well? 3. In L316, could you please elaborate on what "complete/incomplete observations" refers to? 4. L328 recommends Dynamic FDE with Closest, when closed loop simulation is possible. However, why would this be preferable to simply ranking prediction methods by the “Driving Performance” of the planner? 5. Please cite https://arxiv.org/abs/1809.04843 while wrapping up Section 2.2 - this is a well-known study with a similar setup, aimed at planning instead of motion prediction. It would also be interesting to contrast your findings to theirs, given the change in task. 6. Could part of the discrepancy arise from the fact that prediction models penalize errors on all vehicle instances equally, while some are less relevant for planning than others? The result in Table 4 showing the improved correlation when considering only the nearest agents seems to indicate this. 1. Could you add a row with the “Closest” checkmark for the static, multimodal ADE? 2. If it is possible to include a task-aware offline metric such as PKL [15], CAPO [18] or some equivalent in the analysis, this would add significant value to the paper, and I would be happy to increase my rating. Minor: 1. L078: Lastname et al. instead of Firstname et al. Update: Thank you for taking the time to carefully answer my questions, I appreciate the effort. The rebuttal clarifies most of my questions. I understand the choice of Dynamic ADE/FDE for the experiments in this paper (as the rebuttal points out, "a prediction metric that employs the same calculation methodology but behaves distinctively in these two evaluation frameworks" helps provide evidence for their claims). I am still leaning towards a positive rating as I see the main contribution of this paper as highly relevant to the motion prediction community for driving. This is a large research area with hundreds of published papers, and little incentive to move away from static metrics (even though the lack of alignment between open-loop and closed-loop metrics for planning is reasonably well-known). This paper opens up a discussion on the inadequacy of the prevalent motion forecasting metrics, and could be a valuable first step towards more comprehensive evaluation of prediction models in the future.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
4 excellent
Limitations
Limitations are discussed in Section 6.
Summary
The authors examine the discrepancy between trajectory prediction accuracy and driving performance in the task of autonomous driving when a fixed dataset is used and in the presence of multiple surrounding traffic participants (such as other vehicles). Specifically, they assert that this discrepancy arises from: - A dynamics gap between the static dataset that the trajectory predictor is trained on and its subsequent usage for planning in real-world driving scenario. This is because the ego-vehicle’s planner may act differently to the static dataset, which results in novel reactions from the other entities in the environment (vehicles, pedestrians, etc.) and thus out-of-distribution scenarios occur. - Computational efficiency issues caused by the planner and trajectory predictor being too slow to accommodate real-time operation. Although the work does not propose a novel method, the authors conduct of study of existing methods to illustrate the importance of a dynamic approach where the ego-vehicle collects data with the specific predictor (instead of relying on a single fixed static dataset) and efficient trajectory prediction. A wide variety of trajectory prediction models are used (Table 1) to ensure high test coverage along with two planners (RVO and DESPOT) (section 4.2) For trajectory prediction evaluation they use the Average/Final Distance Error (ADE/FDE) in Table 2. To evaluate driving performance a combination of collision avoidance, speed to goal and comfort (minimize jerk) are considered (section 4.4). Experiments first attempt to illustrate how their chosen simulator (SUMMIT, extended from Carla) is suitable to evaluate real-world performance based on simulation (section 5.1, Figure 2) as trajectory prediction simulation results (SUMMIT) and real-world dataset results (Argoverse dataset) are positively correlated. They then illustrate the performance gap that occurs when evaluating against a static dataset versus one collected interactively (dynamically) from simulation where the ego vehicle operates with the predictor (section 5.2, Figure 3. versus Figure 4). Most notably, they find that efficiency concerns are most prevalent under a high planner-simulation tick rate. In this case the predictor efficiency has the greatest effect on driving performance instead of the dynamics gap since it cannot keep up with the high tick rate (Table 3). Table 4 further shows that the discrepancy between driving performance and trajectory prediction accuracy (ADE) is reduced when only nearby agents with full observations is preferred in addition to dynamic data collection.
Strengths
- Experiments are well done and compare a variety of approaches (ten trajectory prediction models, two planners). - Claims are backed clearly by data. The results show the value of using an interactive simulation environment for data gathering and favoring efficient predictors when doing real-time planning for autonomous driving. - In general, well written and clear.
Weaknesses
- The work does not propose a novel approach but is instead of survey of existing methods. - It could also be argued that the survey results are somewhat obvious (which compounds with the first criticism): 1) relying solely on a static dataset for model-based training may result in a distributional shift during real-world evaluation once planning is done, causing failure and 2) efficient predictors are important for real-time planning. Minor Criticism: - The combination of performance metrics $P_{\mathrm{Safety}}, P_{\mathrm{Efficiency}}, P_{\mathrm{Comfort}}$ could be clarified in the text (section 4.4). Sensibly, $P_{\mathrm{Safety}}$ and $P_{\mathrm{Comfort}}$ are to be minimized while $P_{\mathrm{Efficiency}}$ is maximized. But all metrics are labeled “performance” so I was originally expecting them to be handled uniformly. The text does mention that “we normalized the direction of these three metrics” but this is still somewhat ambiguous. Things are fully clarified once looking at the supplemental material where the sign on $P_{\mathrm{Safety}}$ and $P_{\mathrm{Comfort}}$ are reversed during normalization. However, I would find it helpful if this was more clearly mentioned to in the text. - I feel that the first experimental section (section 5.1) is perhaps not relevant to the main purpose of the paper besides illustrating that trajectory prediction accuracy in simulation and the real-world are positively correlated across different models. I imagine most semi-photorealistic simulation environments would possess this property, especially given its extension from the mature Carla simulator. They also do not explicitly show that driving performance is ultimately related (only trajectory prediction performance). There does appear to be a notable ADE/FDE error increase when moving to the real-world dataset. Thus, once applied to planning, I could see a case where a model might perform “well” in simulation but “poorly” in the real world (despite performing better than its peers).
Questions
- Should the minADE and minFDE equations in Table 2 be a function of k? I assume that each predicted $\hat{x}_i$ is a sample taken K times? - In Table 3, is the “Inference Time” for one network inference loop and not the overall time for planning? If it is the overall time for planning I would be confused as to why performance changes for some models below the 30hz refresh rate when going from 30 to 3 to 1 hz. - In section 5.1, lines 237-238 it says that the “nearest three agents are randomly selected to be the interested agent for prediction”. Is there a reason why only 3 agents were considered? Was this the best trade-off between computation time and performance?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
The authors did not explicitly list any limitations of their method. However, their work is primarily a study of existing methods and so a limitations section is perhaps less relevant. Nonetheless, one possible limitation is the focus solely on simulation for illustrating static versus dynamic driving performance discrepancy. Please refer to the weaknesses section for more possible limitations.
Summary
The paper provides an extensive study on the discrepancy between model prediction accuracy and driving performance. It explores two major factors: 1. The dynamics difference; and 2. The computational efficiency of predictors. Various methods are tested in two settings: 1. Fixed Number of Predictions; and 2. Fixed Planning Time. The paper evaluates the model performance on both open-loop prediction and closed-loop diving.
Strengths
The experiments are well-designed and extensive, which provides strong evidence for the statements in the paper.
Weaknesses
While the two factors are well supported by the experiment results given, it is insufficient to say whether other factors are less important, such as the data quality of the open loop eval, the unknowability (future events influencing the ground truth future trajectory) issue for open loop prediction, as well as multimodality in the diving scenarios.
Questions
It would be great if the author could provide add some discussions on possible other factors that cause the discrepancy and why the two factors mentioned in the paper are dominant.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
See questions.
Summary
This work attempts to look for evaluation metrics for trajectory prediction beyond the widely used ADE/FDE metrics. In particular, the authors focus on two overlooked aspects in prediction evaluation, which are: 1) the dynamics gap between the dataset and closed-loop driving scenarios; 2) the trade-off between computational efficiency and performance. The authors argue that we should turn to closed-loop prediction evaluation to address these two problems.
Strengths
This work focuses on a very important and open question for trajectory prediction, which is looking for evaluation metrics for trajectory prediction beyond the widely used ADE/FDE metrics. It is good that the authors emphasize and validate the limitation of open-looped prediction accuracy and the importance of closed-loop evaluation for trajectory prediction.
Weaknesses
1. It is good to see that the authors showed the limitation of open-looped evaluation and the importance of closed-loop evaluation in their experiments. However, I did not think the authors provided any new perspective or insightful solution to the problem of closed-loop evaluation. The value of closed-loop evaluation has been widely acknowledged in the autonomous driving community. It is one of the main motivations behind the emerging research on data-driven traffic simulation. The bottleneck is not the lack of prediction evaluation metrics under the closed-loop evaluation scheme, which is targeted by one of the two claimed contributions of the paper (i.e., dynamic ADE vs. static ADE), but the closed-loop simulation itself. How to synthesize a reliable simulator with realistic reactive agent behavior is an open question. We even do not have a commonly acknowledged set of metrics for evaluating simulation at this stage. Unfortunately, the authors did not discuss the impact of the simulation environment on the closed-loop evaluation result in this paper (e.g., whether the simulated driving performance and the simulated dynamic ADE metric are correlated with real-world driving performance). While the authors adopted different prediction models and planners in their evaluation, they only used a single simulator (i.e., SUMMIT). Moreover, the authors did not include a literature review on traffic simulation and evaluation metrics for traffic simulation. 2. The inherent logic behind the experiments is fundamentally flawed. In Sec. 5.2, the authors showed that static ADE is not correlated with dynamic ADE and driving performance. However, the authors used the correlation of static ADEs on the Argoverse and the Alignment dataset collected from SUMMIT as evidence to validate the realism of SUMMIT. Just like the authors' observation in Sec. 5.2, the correlation in static ADE does not necessarily indicate that the SUMMIT simulation is reliable and realistic under closed-loop simulation and evaluation. To justify the realism of SUMMIT, the authors should adopt those evaluation metrics from the latest literature and compare SUMMIT against those state-of-the-art simulation models.
Questions
1. I do not think the authors appropriately answer the question raised in the title (i.e., what truly matters in trajectory prediction for autonomous driving?). The paper focuses on two perspectives on this question, which are very limited and far from compiling a comprehensive and insightful answer. The experimental setting is also insufficient as an attempt to answer such a bold question (e.g., lacking analysis on the impact of simulation on closed-loop evaluation). 2. It would be beneficial to have some qualitative analysis and visualization beyond statistics. For instance, visualizing some representative examples could be helpful in understanding why static ADE and dynamic ADE are not correlated. 3. The citation for SUMMIT is missing in the main text.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
The authors discussed some limitations and future directions in Sec. 6. However, I think the limitations are not sufficiently addressed. For instance, I would consider the limited evaluation on a single simulator and the missing discussion on the impact of simulation realism as crucial drawbacks of this work.
Summary
In the article, the author discusses the limitations of the current open-loop trajectory evaluation method: 1. It lacks consideration of closed-loop interactions with the ego-planner 2. It overlooks computational timeliness. These two factors result in a misalignment between open-loop trajectory evaluations and closed-loop driving scores. The author also proposes a new metric, Dynamic ADE/FED, which enables a positive correlation to be established between open-loop trajectory evaluations and closed-loop driving scores.
Strengths
* Originality: This paper is original, providing a new evaluation method for the field of trajectory prediction. * Significance: The foundation of this paper is sound. It focuses on analyzing the flaws and issues in the standards of trajectory prediction evaluation, and conducts a substantial amount of experiments across different models. * Quality & Clarity: I believe this section is the weakness of the paper. Please refer to the 'Weakness' part.
Weaknesses
* Clarity & Quality: - 1. The core Dynamic ADE/FDE metrics in this paper are not clearly explained; there is hardly any section dedicated specifically to elaborating on this aspect. It would be beneficial to include an illustration for better understanding. - 2. Despite the numerous experiments conducted, some of them fail to support the arguments made. For instance, on lines 243-245, you stated that the positive correlation demonstrated by different prediction models on the Argoverse and Alignment datasets indicates that there isn't a significant domain gap between the SUMMIT simulator and real-world data. I find this reasoning flawed. A positive correlation merely indicates consistency in the predictive ability of the models across the two datasets; it does not necessarily imply that the behaviors in the two datasets are consistent. Moreover, a closer look at the specific numerical values on the x and y-axes in Figure 2 reveals significant discrepancies between the Argoverse and Alignment datasets. The ADE distribution for Argoverse lies between 1.5-3.5, while that for Alignment falls between 0.5-1.5. This suggests that the Argoverse dataset is substantially more challenging than the Alignment dataset, which contradicts your claim. - 3. Besides this, there are many other similar issues. The overall experimental conclusions lack substantial empirical support and are simply based on statistical analyses of a small number of data points, yielding correlations that aren't particularly strong (the paper only employs eight different prediction models and two different planning models). I remain unconvinced by statistical conclusions drawn from such a limited set of data points. * Novelty: - 4. This paper seems more akin to an experimental report. Despite the multitude of experiments conducted, they remain rather superficial. For a NeurIPS paper, a deeper level of analysis is required, such as quantitative analysis, comparing what specific interactions in which scenarios cause differences between static and dynamic ADE in open-loop and closed-loop evaluations. Additionally, as I previously mentioned, many of the experimental results do not effectively support the conclusions drawn. - 5. The work done in the 'Computational Efficiency' section is quite trivial. The conclusions drawn here are rather obvious, and in the field of autonomous driving engineering, this aspect has already been thoroughly considered.
Questions
Please refer to the 'weaknesses' section.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
3 good
Limitations
Please refer to the 'weaknesses' section.
Thanks to the authors for their detailed response. Please find my follow-up comments to the response below: 1. Novelty: I agree with the authors that the closed-loop evaluation of trajectory prediction is a perspective that has not been directly studied in the literature. However, I still consider the work to have limited novelty, given that closed-loop evaluation has been studied for planning and control [7,8,9] and the limitation of open-loop evaluation has been stated in prior works [4, 5]. 2. Simulation matters for closed-loop evaluation: I disagree with the author's statement that "the impact of the simulation environment is not related to our core contribution". The main difference between dynamic ADE and static ADE lies in evaluating the prediction model over state distribution under closed-loop driving. If the reactive agents' behavior in the simulation is not realistic in those OOD states, how could we trust the computed dynamic ADE metric? Thus, Discussing the effect of simulation to closed-loop evaluation of prediction models is a crucial aspect of the subject the authors aim to study. I understand the current set of results still have their value, but I don't think they are sufficient for a NeurIPS paper. 3. Realism of SUMMIT: In Lines 244-245 of the submitted manuscript, the authors claim that "The consistent results suggest that the Argoverse and Alignment datasets share similar underlying features. Therefore, the SUMMIT simulator can be employed to evaluate real-world performances." I think the second sentence does imply that the SUMMIT simulator is realistic. In fact, related to my last comment, I think evaluating the realism of SUMMIT is necessary in order to validate that the dynamic ADE computed in SUMMIT is reliable. If the authors agree that Fig. 2 cannot validate the realism of SUMMIT, then they should include additional results to justify the realism of SUMMIT. Overall, I still lean towards rejecting the paper in its current form.
>Novelty: I agree ...However, I still consider the work to have limited novelty, given that closed-loop evaluation has been studied for planning and control [7,8,9] and the limitation of open-loop evaluation has been stated in prior works [4, 5]. To address the concerns of novelty, we respond in three aspects. 1. While closed-loop evaluation has been explored extensively for planning and control [7, 8, 9], assuming its equivalent influence on static modules like predictions, without comprehensive experimentation, is unjustified. The inadequacy of such studies has been acknowledged by reviewers as well. In this context, we present comprehensive analysis as the first to undertake this exploration, marking its novelty. 2. We stated that the limitations of the present prediction evaluation have been highlighted in prior works[4, 5]. However, as these papers state, they do not aim to address the inherent limitation of open-loop evaluation, but focus on the asymmetry of prediction error. Despite connecting prediction evaluation with driving performance, they still employ real-world datasets as the standard for evaluating predictors. 3. Conversely, we discover the dynamics gap between current prediction evaluation systems and real-world applications, which is overlooked in previous works. We substantiate this gap with experiments, and demonstrate its strong impact on the correlation between prediction performance and driving performance. As real-world tests are often unaffordable, an alternative approach–closed-loop evaluation– is suggested to ease the dynamics gap. The main contribution of our paper is to reveal such a dynamics gap and its significance, not the closed-loop itself. >Simulation matters for closed-loop evaluation: I disagree with the author's statement that "the impact of the simulation environment is not related to our core contribution"... Thus, Discussing the effect of simulation to closed-loop evaluation of prediction models is a crucial aspect of the subject the authors aim to study. Our results comprise two layers of analysis. The first layer establishes the existence and significance of the dynamics gap concerning the correlation between prediction performance and driving performance. The relationship between simulation datasets and simulation scenarios mirrors that of real-world datasets and real-world scenarios. Our investigation substantiates that the dynamics gap between datasets and real-time systems stands as a foundational factor contributing to the observed weak correlation between prediction evaluation metrics and real-time performance. Crucially, the dynamics gap can solely be effectively mitigated when the predictor is evaluated and applied within the same real-time environment, leading to a substantial correlation between prediction evaluation metrics and driving performance. Our core contribution lies in the analysis of the dynamics gap. This contribution remains unaffected by the realism of the simulator and necessitates real-world tests for the prediction evaluation. The second layer underscores the efficacy of the alternative evaluation protocol (simulation) when real-world tests are unaffordable. To substantiate this, we conduct the Sim-Real Alignment experiment, which serves to establish the alignment between prediction performance of the simulation dataset and real-world datasets. Likewise, the alignment of driving performance is verified via the simulator itself. Thus, we take effort in identifying the optimal simulator, SUMMIT, which is built upon Carla, the most widely-used simulator in recent competitions [6] and research [10,11]. The impact of the simulation environment partially affects the realism of the simulator and offers marginal influence to our contributions. Therefore, we opt not to engage in an in-depth discussion of it within the main text. >Realism of SUMMIT: In Lines 244-245 of the submitted manuscript, the authors claim that "The consistent results suggest that the Argoverse and Alignment datasets share similar underlying features. Therefore, the SUMMIT simulator can be employed to evaluate real-world performances.".... If the authors agree that Fig. 2 cannot validate the realism of SUMMIT, then they should include additional results to justify the realism of SUMMIT. As aforementioned, our core contribution revolves around the concept of the dynamics gap. The closed-loop simulation is served to address this gap only when real-world tests are unaffordable. Even if the simulator is not totally realistic, our core conclusion still stands. The realism of SUMMIT only supports part of our secondary contribution. Besides, although the requirement of realism of the simulator can be eased, we still tried our best to find the best simulator SUMMIT, which is built upon Carla, the most widely-used simulator in recent competitions [6] and research [10,11].
Thank the authors for their follow-up clarification. I do appreciate the authors' effort in identifying the existence and significance of the dynamics gap. I think it is a valid point and the major contribution of this work. However, I don't think this point itself is sufficient to make this paper accepted for NeurIPS, considering that closed-loop evaluation has been studied for planning and control [7,8,9] and the limitation of open-loop evaluation has been stated in prior works [4, 5]. That's why I am suggesting an in-depth discussion on the effect of simulation on the closed-loop evaluation metrics. I agree that the closed-loop evaluation of the prediction model is a promising direction. However, it cannot serve as a sound solution for benchmarking prediction models if the simulation environment is not realistic, even though It could be an acceptable solution for engineering purposes. For example, if one model achieves a smaller dynamic ADE than the other in SUMMIT, does it necessarily mean it is a better prediction model than the other? Since SUMMIT leverages a data-driven prediction model to simulate the reactive agents' behavior, the "ground truth" used for computing the dynamic ADEs can be drastically wrong on OOD states, then the ranking based on dynamic ADE cannot be trusted in this case. After reading all the authors' responses, I am willing to change my rating to 4, which reflects my current impression of this work. Overall, I do think the paper has its value to some degree, but I don't think its contribution is sufficient when considered for NeurIPS.
We are glad that some of your concerns have been addressed and thank you for raising the score! We would like to reiterate that prior works [4, 5] still focus on crafting planning-aware metrics in an open-loop context, instead of closed-loop scenarios. It is crucial to underscore that our contribution mainly stands on providing a fresh perspective to prediction evaluation, highlighting both the presence and equal significance of the dynamics gap and computational efficiency. We expect a highly enhanced correlation with real driving performance using our evaluation protocol, which is realized in our experiments. Since conducting real-world tests is expensive, researchers rely on simulators as an alternative. Simulators have been utilized to conduct numerous research and have achieved significant progress in autonomous driving, e.g., planning and control. In light of that, we opted to employ the CARLA-based simulator, SUMMIT, to enhance the reliability of our results. Thus, while we agree synthesizing a reliable simulator would undoubtedly enhance the credibility of our paper, the current simulator adequately serves to validate our conclusions and provides promising evaluation results. The proposed dynamic ADE notably surpasses current open-loop evaluation metrics, although it is not perfect. We extend our gratitude for your prompt and valuable responses, as well as the time you spent in our discussions.
Thank you for the further elaboration. Some of my concerns have been resolved. I would like to clarify my previous question further: 1. Regarding the consistency issue between Argoverse and Alignment datasets: My intention was not to say that SUMMIT cannot be employed to evaluate real-world performances, but rather to say that your experiment does not adequately reflect your claim. Your designed experiment only shows that different models have positively correlated trajectory prediction performance on these two datasets, which doesn't prove that SUMMIT has the same behavior with the real world scenario. It is possible that SUMMIT and real-world datasets have significant differences in performance, but due to factors intrinsic to the model itself (such as the amount of model parameters, structure, etc.), they show positive correlations in prediction metrics across different datasets. An experiment that could successfully validate your claim might be to demonstrate the consistency of agent behavior in SUMMIT with that of Argoverse agents, such as constructing a SUMMIT scenario that mimics the Argoverse dataset, and then calculating the consistency of agent trajectories in that scenario. Additionally, I have some questions about the starting point of the paper: 1. Since dynamic ADE/FDE requires closed-loop evaluation, why don't we directly use a closed-loop Planning and Control metric, such as Driving Score? 2. Closed-loop evaluation of dynamic ADE/FDE indeed exhibits better consistency with closed-loop metrics, but we can only test it in a closed-loop simulator. This metric cannot be computed on real-world data. 3. Moreover, this metric is influenced by the planner involved in the evaluation; different planners would lead to significant inconsistencies in the dynamic ADE/FDE measurements of trajectory prediction models, as referenced in Figure 4. These factors could have a significant impact on the practical utility of the dynamic ADE/FDE metric.
To answer these questions, we would like to explain more about the starting point of the paper. Our paper comprises two layers of analysis: The first layer establishes the existence and significance of the dynamics gap and computational efficiency, concerning the correlation between prediction performance and driving performance. **The relationship between Alignment dataset and SUMMIT simulator mirrors that of Argoverse dataset and real-world autonomous driving, as each system pair shares identical ego-planner and exo-agents motion model.** The difference in the correlation coefficient between dynamic ADE/FDE and static ADE/FDE versus driving performance is notable, demonstrating the significance of the dynamics gap. Similarly, the importance of computational efficiency is emphasized by the remaining gap between dynamic ADE and driving performance. These two gaps can solely be effectively mitigated when predictors are evaluated in real-world autonomous driving. This layer constitutes our core contribution. The second layer emphasizes the efficacy of the alternative evaluation protocol (simulation) when real-world tests are unaffordable. **To substantiate this, we conduct the Sim-Real experiment, which establishes the alignment of prediction performance between simulation and real-world. Likewise, the alignment of driving performance is verified via the simulator itself.** SUMMIT is built upon Carla, the most widely-used simulator in recent competitions [4] and research [5,6]. The attached motion model GAMMA [9] responds to ego-motion and outperforms that of Carla. Afterward, the ranking of dynamic metrics reflects the ranking of real-world driving performances and is more obtainable. To ensure the protocol's effectiveness across diverse planners, we employ the RVO planner to explore the correlation between dynamic ADE and driving performance. Significantly, a strong relationship between the two persists. >Regarding the consistency between Argoverse and Alignment datasets: I'm saying that your experiment does not adequately reflect your claim. As aforementioned, the Sim-Real experiment ensures alignment in prediction performance, while the alignment of driving performance (the same as behavior) is verified by the SUMMIT simulator along with its affiliated motion model GAMMA [9]. Given our primary focus is not the development of a simulator, our efforts are dedicated to finding the best simulator. >Why don't we directly use a closed-loop Planning and Control metric rather than dynamic ADE/FDE? We agree that the ultimate goal is driving performance. The subtle difference behind using dynamic ADE lies in emphasizing the significance of the dynamics gap. We select a prediction metric that calculates in the same way but exhibits distinct performance in static and dynamic evaluations. In addition, computational efficiency is emphasized by the remaining gap between dynamic ADE and driving performance. The underlying question lies in the significance of dynamic metrics. As aforementioned, our main contributions are: 1. establish the existence and significance of the dynamics gap and computational efficiency. 2. emphasize the efficacy of the alternative evaluation protocol. It's essential to note that dynamic ADE serves solely as the tool we employ to substantiate these contributions, rather than constituting our primary contribution. >We can only test closed-loop metrics in a closed-loop simulator. This metric cannot be computed on real-world data. As aforementioned, the relationship between Alignment dataset and SUMMIT simulator mirrors that of Argoverse dataset and real-world. Thus, dynamic metrics also prove effective in real-world tests. However, they are unattainable from datasets. This underscores our primary finding regarding the importance of real-time evaluation. >Moreover, this metric is influenced by the planner involved in the evaluation...influence the practical utility of the dynamic metrics. This concept forms one cornerstone of our paper: prediction evaluation is dependent on the downstream modules. Previous planning-aware metrics [10] also tried to capture the impact, but in an open-loop manner. It is inappropriate to posit that the 'optimal' predictor will excel 'normal' predictors across all planners. The best predictor for a given planner might vary, depending on its real driving performance. This, naturally, leads to corresponding changes in the ranking of dynamic ADE. Therefore, our main contributions are: 1. identifying the importance of dynamics gap and computational efficiency 2. highlighting the efficacy of the alternative evaluation protocol, rather than proposing the dynamic ADE. We extend our gratitude to the valuable insights and constructive feedback, which greatly enhance the quality of our manuscript. [9]Luo et al. Gamma: A general agent motion model for autonomous driving [10]Ivanovic et al. Injecting planning-awareness into prediction and detection evaluation
Thank you for responding to my comments. I appreciate that this work may give a more comprehensive discussion of relevant metrics to indicate actual driving performance after training (static versus dynamic ADE, computational performance) but I still currently feel that the level of contribution and novelty is not quite sufficient for NeurIPS. I imagine static datasets will still be used in training for quite some time (ex: for real-world applications) and so I would of found it more useful if ways to address this misalignment were proposed. The proposed metrics do more strongly indicate actual driving performance, but they also require direct evaluation in an interactive simulation environment and pairing with a planner and so I am not sure of their usefulness as a final evaluation metric instead of simply using driving performance itself, although they can act as a useful tool to indicate why driving performance may be poor. But as a tool for indicating why driving performance may be poor, these again seem like obvious metrics where dynamic ADE is essentially the same prediction error computed during evaluation (a sort of test set error) and computation time is clearly relevant for real-time planners. I agree that closed-loop metrics are often overlooked when evaluating the motion prediction model but I am unsure if this is due to the fact that they are unknown or simply to do with the hurdles in integrating them into the evaluation pipeline: they require an interactive simulation environment (difficult for real-world data) and pairing with a planner from which results may vary from planner to planner. Or is it possible that since the prediction models are now being evaluated on different datasets as an outcome of their interaction with the environment, the comparison is no longer fair (i.e. one dataset is harder than the other)? Thus I am still somewhat unsure of the level of contribution of this work.
We thank the reviewer for engaging in the discussion! Regarding concerns about the novelty of our paper, it is essential to emphasize that the dynamics gap is not a sort of test set error. **The dynamics gap is the inherent limitation of open-loop evaluation, in which exo-agents will not react to the change of predictor and corresponding ego-motion**. Conversely, in the real-world, different predictors result in varied behaviors of the ego-agent, which, in turn, influence the future behaviors of other road users, leading to different dynamics within the environment. Without addressing this gap, all prediction methods remain limited in real-world utility. While we agree that datasets will still be used for training purposes in the future, a shift towards dynamic datasets will happen, rather than relying on static datasets. Dynamic datasets (or offline simulators) like MixSim [8] and nuPlan [9] are promising to replace the role of current open-loop datasets. Researchers have the opportunity to train and evaluate their models using interactive agents, instead of static records. However, the awareness of the urgency to evaluate predictors within closed-loop remains limited. These dynamic datasets are all designed for evaluating planners. Besides, there is no comprehensive study about the limitation of current open-loop evaluation, and the key factor leading to it. Although dynamic datasets have yet to be employed for prediction evaluation, the reveal of dynamics gap can still make substantial contributions to the autonomous driving field. We present a potential application here: motion model evaluation. Motion Model Evaluation: While each simulator employs its unique motion model for exo-agents, the assessment of motion model quality remains challenging. Through the utilization of dynamic ADE, three values can be computed utilizing the same predictor and planner: static ADE within the simulation dataset, dynamic ADE within the simulation environment, and dynamic ADE during real-world testing. These metrics solely involve changes in the motion model of exo-agents, enabling the evaluation of the simulation's fidelity to real-world agents' motion. e.g. $$ fidelity =\frac{ADE_{real} - ADE_{sim}}{ADE_{real} - ADE_{static}} $$ Again, we would like to thank the reviewer for thoughtful and comprehensive comments. We would be more than glad to discuss any remaining concerns you might have. [8] Suo et al. MixSim: A Hierarchical Framework for Mixed Reality Traffic Simulation. CVPR 2023. [9]nuPlan Planning Challenging: Closed-loop reactive agents, 2023
To clarify: When I likened the dynamic ADE to a "sort of test set error," I did not mean to imply an exact equivalence to a conventional static test set in the machine learning context. This was not the main point of my comment. I realize that evaluation in a closed-loop interactive environment after open-loop training is not a conventional fixed test set. It changes based on the trained motion prediction model paired with a planner and the reactions of other agents in the scene to the ego vehicle, and this introduces additional complexities. I mentioned this in my original reply. The purpose of my comment was to point out that using dynamic ADE as a tool to diagnose the dynamics gap seems like an obvious check to see if things are working correctly during evaluation (i.e. like thinking to measure the test set error). Given that these metrics (re-calculating ADE during closed-loop evaluation and computation time) seem relatively self-evident I feel this reduces the impact of the work. I also feel that the interplay findings between dynamic ADE and computation time affecting driving performance is also somewhat obvious (with computation time being more relevant to driving performance when the planning time budget is tight). If the main purpose of the work is also to make the performance gap issue known then I also feel that this lacks a high level of novelty since other works have also pointed out the discrepancies in performance that arise from open-loop training and closed-loop evaluation giving rise to out-of-distribution scenarios. I also question if the lack of closed-loop evaluation is always related to a lack of knowledge that this dynamics gap exists (that this work is trying to make known) or simply other hurdles in the evaluation pipeline instead. For example: closed-loop evaluation may not be possible for real-world datasets, or variability introduced when paired with different planners. I agree that addressing this performance gap is an important area of future research. But I am unsure if simply pointing out the issue (which already seems known from previous works) with (what I consider obvious) metrics to evaluate it meets the contribution requirements for NeurIPS. I feel the work has value: it is an important issue to discuss, experiments confirm the claims made, and it gives greater experimental detail as to why this gap occurs and have increased my contribution score slightly to reflect this. But I unfortunately feel that the contribution level is still not significant given my previous comments.
We divide the questions into two aspects: **the awareness of dynamics gap**, and **the reason behind the absence of reactive evaluation research**. For the first question, it is mentioned that the dynamics gap appears rather self-evident, thus reducing the impact of our work. However, the 'self-evident' limitations of current prediction evaluation are numerous: awareness of drivable area, near collision, social conventions, etc. Nevertheless, quantitative analysis is missing to determine the priority of addressing these limitations and to find the dominant factor on ultimate driving performance. An illustrative instance is the asymmetry of prediction error [10]. Even when two prediction models demonstrate identical ADE values, their driving performances will not align due to the varying impact of prediction error on ego-plan. This factor may appear self-evident, and the challenge lies in assessing its relative significance. In our experiment, we account for the most critical aspect of asymmetry prediction error by comparing "Closest Dynamic ADE" with "Dynamic ADE." Consequently, the influence of asymmetry on the correlation between ADE and Driving Performance is **much smaller** than that of dynamic gap. (Influence For RVO: 0.16 vs 0.61; Influence For DESPOT: 0.20 vs 0.96) **Our contribution goes beyond mere revealing these factors. We offer a comprehensive quantitative analysis to uncover the dominant role of the dynamics gap in shaping real-world driving performance**. Unlike other factors, the development of prediction methods will encounter significant limitations if the dynamics gap remains unsolved. Upon addressing the first question, the response to the second question becomes evident: researchers have yet to recognize the significance of the dynamics gap. The common belief is that such a dynamics gap exerts marginal influence on evaluation and is not worth additional training and simulation to tackle. On the contrary, our experiments demonstrate that the dynamics gap holds a dominant influence on driving performance, which is the starting point of our paper. In addition, since the optimal solution to planning-aware prediction evaluation is to integrate it with a real planner, pairing with planners is not a drawback of our work. Considering the variation of planners may lead to concerns regarding fair prediction evaluation, a uniform cost function can be designed for all planners to form a fair comparison. As the discussion phase draws to a close, we would like to extend our gratitude once again for your prompt response and valuable insights. [10]Ivanovic, Boris, et al. "Injecting planning-awareness into prediction and detection evaluation." IEEE IV, 2022.
Thank you for taking the time to carefully answer my questions, I appreciate the effort. The rebuttal clarifies most of my questions. I understand the choice of Dynamic ADE/FDE for the experiments in this paper (as the rebuttal points out, "a prediction metric that employs the same calculation methodology but behaves distinctively in these two evaluation frameworks" helps provide evidence for their claims). However, I am still missing an answer as to whether the Dynamic ADE/FDE has any benefit as a metric to a practitioner building prediction systems for driving scenes. The same question has been raised by reviewer Aygk, and I am curious to see the answer. I am still leaning towards a positive rating as I see the main contribution of this paper as highly relevant to the motion prediction community for driving. This is a large research area with hundreds of published papers, and little incentive to move away from static metrics (even though the lack of alignment between open-loop and closed-loop metrics for planning is reasonably well-known). This paper opens up a discussion on the inadequacy of the prevalent motion forecasting metrics, and could be a valuable first step towards more comprehensive evaluation of prediction models in the future.
Thanks for your recognition of our work. In this paper, our main contribution stands on: 1. Establish the existence and significance of the dynamics gap and computational efficiency. 2. Emphasize the efficacy of the alternative evaluation protocol (simulation) when real-world tests are unaffordable. In this perspective, dynamic ADE/FDE serve solely as the tools we employ to substantiate these two contributions, rather than constituting our primary contribution. The difference in the correlation coefficient between dynamic ADE/FDE and static ADE/FDE versus driving performance is notable, demonstrating the significance of the dynamics gap. Similarly, the importance of computational efficiency is emphasized by the remaining gap between dynamic ADE and driving performance. However, the benefit of dynamic ADE to a practitioner building autonomous driving systems is also significant, as the primary step in solving dynamic gap is to perceive it. Dynamic ADE versus static ADE can serve as a metric to value how much the dynamics gap is solved. We present a potential application here: motion model evaluation. Motion Model Evaluation: While each simulator employs its unique motion model for exo-agents, the assessment of motion model quality remains challenging. Through the utilization of dynamic ADE, three values can be computed utilizing the same predictor and planner: static ADE within the simulation dataset, dynamic ADE within the simulation environment, and dynamic ADE during real-world testing. These metrics solely involve changes in the motion model of exo-agents, enabling the evaluation of the simulation's fidelity to real-world agents' motion. e.g. $$ fidelity =\frac{ADE_{real} - ADE_{sim}}{ADE_{real} - ADE_{static}} $$ We would like to thank the reviewer for thoughtful and positive comments. We would be more than glad to discuss any remaining concerns you might have.
Follow-up question
Thanks again for the clarification. I concur on the significance of the 2 main contribution points, and I think this paper provides evidence to the community that evaluating prediction models through simulation is worth the effort. While the motion model evaluation idea is interesting, it remains secondary to the main contributions, so I would suggest mentioning it in the supplementary material. I have one final follow-up question: will the code to reproduce your experiments be made available? If so, am I correct in understanding that with access to this code, future work on motion prediction could plug in their predictors to this code and be able to report Dynamic ADE on SUMMIT? Given the mixed reviews, I believe that the paper could have done a better job at emphasizing the significance of the main contributions. Nevertheless, I think the discussion phase will help the authors better present this point, and I lean towards maintaining my initial rating of "Accept".
Thanks for your active engagement in the discussion. We deeply appreciate your thorough review of all the comments, as well as the valuable insights and recognition regarding our core contributions. We fully concur that highlighting the dominant influence of dynamics gap and computational efficiency on driving performance is crucial in our main text, as it distinguishes our work from previous studies within the field. The motion model evaluation idea and Sim-Real Alignment experiment should be moved to the supplementary material. For the follow-up question, code release is currently in progress. We have tidied up the code for predictor implementation and scenario analysis. Given the complexity for simulation with planners, we have opted to leverage Docker to facilitate future application of our work. This may take some additional time. We aim to publish our code soon to provide an interactive evaluation protocol for prediction methods.
Decision
Accept (poster)