Learning Cooperative Trajectory Representations for Motion Forecasting

Summary

This paper introduces V2X-Graph, a novel framework using trajectories of agents (include ego and others) and vector map as inputs for cooperative motion forecasting that fuses trajectory features in an interpretable, end-to-end manner. The authors evaluate V2X-Graph using V2X-Seq, a vehicle-to-infrastructure (V2I) motion forecasting dataset. Additionally, they create V2X-Traj, the first real-world dataset for vehicle-to-everything (V2X) motion forecasting, including scenarios with multiple autonomous vehicles and infrastructure. Author shows the V2X-Graph is achieving state-of-the-art results on both V2X-Seq and V2X-Traj datasets.

Strengths

1) The paper is well-written, with a clear introduction of methods and detailed explanations of the experiments conducted. 2) This paper proposes a novel framework to address motion forecasting in V2X scenarios that not only encodes multiple trajectories with map information but also designs additional graphs for heterogeneous feature representations. Moreover, the paper discusses the effectiveness of different components. 3) This paper introduces the V2X-Traj dataset, which includes rich information such as trajectories, traffic lights, and maps, making it a valuable resource for the community to study V2X scenarios.

Weaknesses

The proposed method emphasizes V2X; however, the inputs do not include important factors such as traffic lights, lane markings, and pedestrians, which may significantly impact the future motion of the vehicle. There is concern that V2X-Graph oversimplifies the problem.

Questions

1. There are more works of GNNs for motion forecasting such as [1][2][3]. 2. What’s the efficiency or latency of this framework given it has quite complicated network components? 3. Please check the weakness. [1] Mohamed, Abduallah, et al. "Social-stgcnn: A social spatio-temporal graph convolutional neural network for human trajectory prediction." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020. [2] Li, Jiachen, et al. "Evolvegraph: Multi-agent trajectory prediction with dynamic relational reasoning." Advances in neural information processing systems 33 (2020): 19783-19794. [3] Girase, Harshayu, et al. "Loki: Long term and key intentions for trajectory prediction." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The paper discusses the limitations of the work in section 6.

Summary

This paper presents the V2X-Graph method for cooperative motion forecasting. In a cooperative autonomous driving setting, an autonomous driving vehicle receives sensor data from surrounding vehicles and infrastructure-side devices. Existing cooperative autonomous driving approaches focus on perception completion, where the vehicles receive only perception data, and motion forecasting is performed with completed perception data without considering the cooperative setting. The proposed V2X-Graph method is a graph-based framework that performs cooperative motion forecasting with trajectory feature fusion. It fuses history trajectories from different views with an Interpretable Graph. The Interpretable Graph consistents of three subgraphs: Motion Fusion subGraph (MFG), Agent-Lane subGraph (ALG), and Cooperative Interaction subGraph (CIG). The authors also provide a new dataset V2X-Traj. Compared to the existing V2X-Seq dataset which contains only vehicle-to-infrastructure (V2I) cooperative scenarios, V2X-Traj also includes vehicle-to-vehicle (V2I) cooperative scenarios, making it vehicle-to-everything (V2X). The authors evaluated their V2X-Graph method on the existing V2X-Seq dataset as well as their new V2X-Traj dataset. They compared against the PP-VIC baseline. The result shows that adding trajectory feature fusion improves the performance over the PP-VIC baseline.

Strengths

* The proposed V2X-Graph method improves the cooperative motion forecasting performance over the PP-VIC baseline. * The new V2X-Traj dataset will be very useful to the community.

Weaknesses

* I am not fully convinced by the motivation of why perception completion is not enough for cooperative motion forecasting. It will be useful to give some examples to explain how feature fusion helps. * The writing can be improved. For example, it will be useful to give an overview of the architecture either in the caption of Figure 2 or at the beginning of Section 4. The module names MFG, ALG, CIG, etc in the architecture diagram are not explained until Section 4.2. The evaluation setup of Table 2 (Graph-based methods comparison) is not clearly explained.

Questions

* Table 1: Why are the PP-VIC + HiVT results different from those in the original PP-VIC paper? * Line 284: How are the observations of the same agent from different views aggregated?

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

3

Limitations

Yes

Summary

This paper tackles the cooperative motion forecasting problem for vehicles. This paper introduces additional information other than the ego view agent from other view to expand the perception field of the prediction. The authors propose a graph network to extract information for multimodal trajectory prediction. The authors introduce a new dataset, namely V2X-Traj, for this problem setting. Experiments show the efficacy of the proposed method.

Strengths

1. The problem setting is new. With more and more autonomous vehicles on the road, it is possible to include cooperative information. 2. The experimental results on both datasets show that V2X-Graph outperforms existing methods.

Weaknesses

1. The cooperative setting is limited to two vehicles. 2. The scalability of the V2X-Graph framework in terms of computational resources and its performance with larger datasets or in more complex environments is not extensively discussed. The training cost and inference cost should be compared with the baselines. 3. There is not error analysis. When does the method/cooperative setting fail and why? 4. Potential cooperative forecasting problems are not considered. How does the V2X-Graph framework handle communication delays or data loss in real-time applications?

Questions

Please address the weaknesses. ---Post rebuttal Thank you for the author's response. The reply addressed most of my concerns and the added experiments on inference efficiency are important. I will increase my score

Rating

5

Confidence

2

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors have discussed the potential limitations.

Summary

The paper introduces a novel graph-based framework called V2X-Graph for learning cooperative trajectory representations in motion forecasting for autonomous vehicles. V2X-Graph aims to enhance the motion prediction capabilities of autonomous vehicles by leveraging cooperative information from vehicles and traffic infrastructure. The framework represents trajectories as nodes in a graph and fuses motion and interaction features from multiple views to achieve interpretable end-to-end trajectory feature fusion. The authors also constructed the first real-world V2X motion forecasting dataset, V2X-Traj, which includes cooperative scenarios with multiple autonomous vehicles and traffic infrastructure. Experiments conducted on both the V2X-Seq and V2X-Traj datasets demonstrate the advantages of V2X-Graph in utilizing additional cooperative information to enhance motion forecasting capabilities.The creation of the V2X-Traj dataset, a real-world, public dataset for V2X motion forecasting, adds significant value to the field, promoting further research in this domain.

Strengths

- The paper introduces a novel forecasting-oriented representation paradigm, V2X-Graph, which leverages cooperative perception information from multiple sources (e.g., infrastructure and other vehicles) to enhance motion forecasting for autonomous driving. This approach is unique in focusing on the holistic historical motion and interaction features rather than single-frame perception. - The methodology section (pages 4-6) is detailed and provides a comprehensive description of the V2X-Graph framework, including scene representation with graphs and trajectory feature fusion. The use of graph neural networks (GNNs) to encode cooperative scenarios and perform motion forecasting is well-justified. - The paper includes extensive experimental results (pages 8-10) on both the V2X-Seq and V2X-Traj datasets, demonstrating the effectiveness of the proposed approach with clear performance improvements over baseline methods. - The paper is well-structured, with a clear flow from the introduction of the problem (pages 1-2), through the related work (pages 2-3), to the proposed methodology and experimental results. Each section builds logically on the previous one, making the paper easy to follow. Figures and tables (e.g., Figure 1 on page 3 and Table 1 on page 8) are effectively used to illustrate key concepts and results, enhancing the reader’s understanding. - By addressing the underutilization of cooperative perception information in motion forecasting, the paper tackles a critical challenge in autonomous driving. The proposed V2X-Graph framework has the potential to significantly improve the safety and efficiency of autonomous vehicles by leveraging a richer set of perception data.

Weaknesses

- While the application of graph neural networks (GNNs) to encode cooperative scenarios is well-executed, the approach may lack significant novelty. GNNs have been widely applied in various fields, including autonomous driving. The paper would benefit from a more detailed comparison with existing GNN-based methods for motion forecasting to highlight the unique contributions and advancements of the proposed V2X-Graph framework (page 6, Section 3.2). - The experimental evaluation compares the proposed method primarily against a few baselines. Including a broader range of state-of-the-art methods for motion forecasting, especially those utilizing cooperative perception, would strengthen the validation of the proposed approach. This would provide a more comprehensive view of where V2X-Graph stands in the context of current research. - Limited Discussion on Real-World Implementation: lacking a thorough discussion on the practical challenges and considerations of implementing the V2X-Graph framework in real-world autonomous driving systems. Issues such as communication latency, data synchronization, and robustness to sensor failures are crucial for practical deployment but are not sufficiently addressed (page 10, Section 5, Discussion). - Some parts of the methodology section are dense and might be challenging for readers to follow, particularly those less familiar with GNNs. Simplifying the mathematical notation or providing additional explanatory text and illustrative examples would improve clarity and accessibility

Questions

While the experiments demonstrate the effectiveness of V2X-Graph on the V2X-Seq and V2X-Traj datasets, these datasets may not capture the full range of scenarios encountered in real-world driving. Expanding the evaluation to include more diverse and challenging driving environments would enhance the generalizability and robustness of the proposed method. V2X-Traj dataset, while a valuable contribution, might have inherent biases that are not discussed in the paper. Providing a more detailed analysis of the dataset's composition, including the diversity of scenarios and the representativeness of different driving conditions, would be beneficial (page 7, Section 4.1).

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

- The paper briefly mentions some limitations in the discussion section, but it does not provide an in-depth analysis. The authors should more thoroughly acknowledge the limitations of their approach, including potential biases in the V2X-Traj dataset and the constraints of their methodology. Specific limitations such as the scalability of the V2X-Graph framework in highly dynamic and dense traffic environments, potential issues with data communication latency, and the synchronization of information from multiple sources are not adequately discussed. Including a more detailed assessment of these factors would be beneficial. Others: - The paper should address practical challenges such as the computational overhead required for real-time processing, the robustness of the system in adverse weather conditions, and the reliability of V2X communication in various urban and rural environments. Discussing potential solutions or future work aimed at overcoming these challenges would strengthen the practical applicability of the proposed method. - While the paper makes a significant contribution to the field of cooperative motion forecasting, it would benefit from a more thorough discussion of its limitations and potential negative societal impacts. By addressing these aspects in detail and providing constructive suggestions for mitigating these issues, the authors can enhance the robustness, and applicability of their work.

Response

Thank you for your detailed rebuttal and for addressing my previous comments. While I appreciate the efforts taken to differentiate your work from existing approaches, I still have concerns about the novelty of the GCN approach used in the paper. To better understand the contributions and improvements over existing GCN methods in the motion prediction domain, I would like to request the following: - Comparison with Existing GCN Methods. Please provide a detailed comparison of your GCN approach with other GCN methods specifically used in the motion prediction domain. Highlight the specific differences in the architecture, edge encoding, and feature propagation mechanisms. This comparison should include a discussion on how these differences lead to improved performance or new capabilities. - Improvements Over Existing Methods. Discuss the specific improvements your GCN method brings over existing GCN methods in terms of accuracy, interpretability, and computational efficiency. Include quantitative metrics or qualitative analyses that support these claims. - Tailored Design for V2X-Traj Dataset. Given the unique characteristics of the V2X-Traj dataset, I would like to know if the GCN architecture was tailored to leverage the specific properties of this dataset. For example, does the GCN architecture take into account the unique cooperative nature of the data and the varying levels of interaction between agents? If so, please describe these design choices and how they contribute to the effectiveness of the method. - Insight into Design Choices. Provide deeper insights into the design choices made for the GCN architecture, particularly in relation to the V2X-Traj dataset. This includes discussing the rationale behind the choice of edge encoding, node feature extraction, and the inclusion of graph link prediction. How do these design choices specifically address the challenges posed by the cooperative motion forecasting task?

Official Comment by Authors

Dear Reviewer w4tS: Thanks for your response to our rebuttal. In the rebuttal, we primarily made clarifications to potential misunderstandings regarding the formulation of the problem and provided experimental results addressing your concerns about computational efficiency. We are very glad that our response and experimental results have addressed your main concerns and meet your expectation. On that basis, will you raise your rating, or is there any other concern that needs further clarification? We look forward to your ratification and further valuable discussions :). Best Regards,\ 4569 Authors.