SMART: Scalable Multi-agent Real-time Motion Generation via Next-token Prediction

Summary

The main contributions of the paper include: 1) proposing a new motion generation framework that combines road and proxy trajectory labeling schemes with a decoder only Transformer for training the next token prediction task; 2) The model demonstrates zero sample generalization ability and scale law across different datasets; 3) SMART has achieved cutting-edge performance in most indicators, and the single frame inference time is controlled within 15 milliseconds, meeting the real-time requirements of interactive simulation for autonomous driving.

Strengths

SMART introduces a novel method for generating autonomous driving actions, which converts map and trajectory data into sequence tokens and uses a Transformer structure for prediction, achieving effective modeling of real driving behavior. In terms of multiple metric, SMART ranks among the top in the Waymo Open Sports Dataset ranking, especially in terms of inference speed, demonstrating efficient performance and zero sample generalization ability.

Weaknesses

- The scale law ability of models based on transformer structure has been proven in many papers, resulting in limited novelty of the method proposed in this paper. - The result in "Table 4: Absorption study on each component of SMART" indicates that "RVT", "NAT", and "NRVT" are harmful for models trained on WOMD.

Questions

- can the author provide results on NuPlan and compare with the latest works?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The paper explores motion prediction in GPT-style network, which has been done in many works such as StateTransformer. I recommend that the authors should include more discussion about the differences and improvements compared to the previous works.

Summary

In this paper, a GPT-style motion generator is developed for scalable multi-agent simulation. Though various techniques such as motion tokenization, factorized agent attention, and next map segment prediction, the proposed SMART framework ranked 1st place on WOSAC leaderboard for meta metric. Further zero-shot and scalablity analysis further manifest the generizability for proposed SMART framework.

Strengths

1. A unified token design for motion and map, with straightforward decoder-only GPT structure in agent simulation. 2. Solid performance on WOSAC leaderboard for meta metric over other sim agent methods. 3. Comprehensive analysis on scalabllity and zero-shot transferrability.

Weaknesses

1. Minor in methodology differentiating with Trajeglish[1] and MotionLM[2], other auto-regressive decoder for motion prediction or sim agent. 2. Lack of RoadNet evaluation on the road-vector NTP task. 3. Minor in the experimental details, such as clearer auto-regressive sampling process, sim performance for different scale, etc. 4. A thorough check for notations and writing. For instance, I found a suspected ChatGPT sentence: "Here’s the revised version of your text with improved precision and grammar:" in line 508; Also, the notation for modality should be replaced to differentiate with "Value" Reference: [1] Philion, J., Peng, X. B., & Fidler, S. (2024). Trajeglish: Traffic Modeling as Next-Token Prediction. In The Twelfth International Conference on Learning Representations. [2] Seff, A., Cera, B., Chen, D., Ng, M., Zhou, A., Nayakanti, N., ... & Sapp, B. (2023). Motionlm: Multi-agent motion forecasting as language modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 8579-8590).

Questions

1. Whats the difference in decoder design compared with Trajeglish? Please clarify. 2. What is the defination and learning process for road-vector NTP? Whats the performance of SMART without this augmentation? 3. How is the inference process for SMART? Does any sampling trick conducted in SMART?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

N/A.

Summary

This paper presents SMART, a model for multi-agent traffic simulation. The approach is based on a decoder-only transformer architecture that predicts all agents motion tokens autoregressively over time. The architecture makes use of factorized attention layers (over map, agents, and time) and relative positional encodings between agent and map tokens. As in Trajeglish, SMART uses the K-disk algorithm to tokenize agent motion trajectories. However, SMART also tokenizes road vectors and does next-token prediction for road tokens as well for pre-training. Experiments on the Waymo Open Sim Agents Challenge shows that SMART achieves state-of-the-art performance. In addition, SMART's tokenization strategy allows it to generalize better between datasets (NuPlan and WOSAC).

Strengths

- This paper presents a simple architecture for multi-agent traffic simulation that achieves state-of-the-art results in WOSAC. The architecture is quite similar to Trajeglish. However, SMART uses a decoder-only architecture, also tokenizes the road vectors, and uses factorized self-attention with relative positional encodings. In the paper, the authors demonstrate the efficacy of tokenizing road vectors for generalizing between datasets. - The scaling law and zero-shot generalization experiments are interesting. To my knowledge, they are the first experiments of their kind in multi-agent traffic simulation. The zero-shot experiments also demonstrate the efficacy of tokenizing road vectors and using noised tokens for data augmentation during training, which validate the authors' design choices and provide interesting insights for others in the field. - The ablation studies offer interesting insight into SMART's design choices; e.g., Table 7. - Code is available as a part of the submission. - The paper is generally well-written and easy-to-follow.

Weaknesses

- Without comparisons to other methods, it is difficult to interpret the significance of the scaling law and zero-shot generalization experiments. For example, while it appears that SMART's performance scales with its number of parameters, we cannot conclude whether it does so better than other architectures. Likewise, while SMART can zero-shot generalize from NuPlan to WOSAC, we cannot conclude whether it does so better than other architectures. This limits the significance of these experiments. - The paper's results can be made stronger with more statistically significant experiments. For example, in Table 9, it is unclear whether the differences between SMART 8M, 36M, 96M are statistically significant or within the bounds of noise, given how close the numbers are.

Questions

1. How much does road vector next token prediction contribute to SMART's performance? 2. In Table 1, why is SMART's minADE higher than that of Trajeglish? This doesn't seem to match the results in Table 6 of the appendix. 3. Table 4 shows the efficacy of certain architecture choices on generalizing from one dataset to another. How do these architecture choices affect the model's ability to learn from multiple datasets? Do these choices matter in the setting where you have access to NuPlan, WOSAC, and the proprietary dataset? 4. L308: Why does dataset size limit your architecture size to 100 million parameters? Is this a matter of overfitting? If so, I think that it would be interesting to make note of this in the appendix of your paper. Other: - In L508, there may be an unintended sentence "Here’s the revised version of your text with improved precision and grammar:"

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper adequately addresses limitations.