SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rollout

Summary

This paper provides a diffusion-based method for traffic simulation, called SceneDiffuser. SceneDiffuser unifies simulation, including scene initialization and scene rollout for generating initial scene layouts and simulating closed-loop agent behavior. To accelerate the time-cost diffusion process, this paper introduces an amortized diffusion technique to accelerate the process with 16x fewer inference steps. The paper further introduces LLM for few-shot prompt engineering. For verification, the SceneDiffuser achieves good performance on Waymo Open Sim Agent Challenge, top performance on both open-loop and close-loop setting.

Strengths

1. I think it is good introducing diffusion mechanism to the topic of traffic simulation. Previous methods, like MTR++ or TRAJEGLISH typically use transformer architecture and formulate it as a regression or auto-regressive regression problem. 2. The extension to LLM looks interesting to me.

Weaknesses

1. I think the writing can be improved. a. It is hard to understand Algorithm 1-3 given so many details and notations without explanation. b. The role of Figure 4 is to illustrate the auto-regressive process, or not? It confuses me when reading the figure. c. The amortized diffusion part is hard to understand. 2. The motivation of the LLM extension is unclear to me. For me, it seems like an application of the SceneDiffuser, and has nothing to do with the improvements of SceneDiffuser to existing traffic simulators. a. It would be interesting to see how LLM + SceneDiffuser helps autonomous driving. 3. Missing experiments: a. How about the proposed method compared to SceneTransformer [1], especially in terms of performance. For me, I think the two papers share some insights and should be compared fairly. [1]. Scene Transformer: A unified architecture for predicting multiple agent trajectories

Questions

Please see weaknesses.

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

N/A

Summary

The paper proposes a diffusion model data driven simulation of driving scenes. The model is able to both generate driving scenes and to perform closed loop simulation of these driving scenes. The problem is posed as an inpainting task in the scene tensor, which contains all the agents, time steps (past and future) and road user attributes/states. To avoid excessive compute when using the diffusion model in a receding horizon fashion (autoregressive rollout) the paper proposes to reuse the solution of the last iteration (similar to nonlinear MPC) and use a custom noiseing strategy that adds more noise to time steps further in the future. The resulting solution is competitive with other diffusion based methods at a fraction of the compute and reusing the previous trajectory as a warm start allows for more consistent trajectories over time.

Strengths

The computational burden of diffusion policies is in my opinion one of their main drawbacks. The proposed amortized diffusion allows to reduce this burden while resulting in more consistent trajectories over time. By formulating the problem as an inpainting problem the method is general and can tackle several interesting tasks.

Weaknesses

- By relying on the previous solution, there seems to be a risk of getting stuck in local minima. Did you notice such situations? - Implementing hard constraints by projection seems risky, especially in the amortized rollout phase where only one denoising step is taken. I can imagine that for some types of constraints this works well but others will be problematic. Similar to optimization problems where bounds can be handled with clipping but more general constraints need more elaborate implementations (e.g. barrier functions in interior point methods). Did you notice that generalized hard constraints worked better or worse for some types of constraints?

Questions

- How is it enforced that the size and type of a vehicle do not change over time? - Did you investigate taking several denoising steps in the rollout phase. In nonlinear MPC this can often improve the results if the computation load allows it and reduce the dependency on the previous trajectory. - Why is the offroad rate not lower, is it not possible to constrain the motion to onroad predictions? - What is the run time of the model in the rollout phase, time for one denoising step? - The WOMD setup has a horizon of 8s, in the rollout phase do you always predict the full 8 seconds or do you reduce the horizon length. If you keep the 8s horizon how do you deal with the horizon potentially leaving the map. Similarly if you would reduce the prediction horizon to about 4s how would this change the results? Typo: - L261: notedly - notebly - L18: We demonstrate of effectiveness - We demonstrate the effectiveness

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations have been adressed

Summary

The paper proposes a method called SceneDiffuser, to generate multi-agent scenarios for autonomous driving and rollout the scenarios. The two tasks are unified in a model by formulating them as an inpainting task for the scene tensor. The diffusion model uses an amortized diffusion technique to align the diffusion steps with the physical prediction timestamp, enhancing the efficiency. Additionally, language-based agents can be inserted in the scene tensor and rule-based constraints are applied in the diffusion procedure. The scene rollout task is evaluated on the Waymo Sim Agent task and achieves comparable results with SOTA auto-regressive methods, and the scene generation task is validated on the Waymo Motion dataset to realize great realism.

Strengths

- Unifying the scene generation and simulation rollout in a single model is great and relatively new. As far as I know, some models can use post-processing to realize these two closely related tasks while there are no unified models previously. - The figures are very illustrative and can help understanding. The paper is easy to understand in an overall sense. - SceneDiffuser adopts the amortized diffusion strategy from the human animation literature. It eases the efficiency problem in simulation rollout. - The overall performance is great. Scalability is validated as well. Hard constraints are injected during diffusion, to solve the problem of realism.

Weaknesses

I do not see critical problems, but there are a few things worth discussion and improvement. - Metrics. It lacks descriptions or some simple introductions about the metrics for those experiments. For example, the statement in Line 234 ("different metrics buckets are aggregated per-scene instead of per-agent") could be further elaborated for better readability. The scene generation task uses some different metrics, compared to previous scene generation methods, eg [8,a]. Detailed introductions and discussions on these issues would be ideal. - The layout and the order of the tables and figures, can be further improved. Figures and tables are not ordered by the reference order and are placed close to where they are referenced. I acknowledge that it could be difficult to perfectly place them considering the amount and different sizes. However, the current layout and order pose difficulties in reading the paper, making the reading process not smooth. - Though the paper focuses on the scene generation and simulation rollout, similar to previous related literature. If the generated scenes can be utilized to help motion prediction or even other downstream tasks, the impact of the paper would be broader. - There are some related works about scene generation not discussed, eg [a-c]. I am also wondering if those using WMOD could be compared quantitatively. This question also relates to the metrics. - The proposed method still can have realism problems, even though constraints are injected in the diffusion procedure. This internal drawback of diffusion methods could not be fully relieved. - Typos: - Line 18: We demonstrate - Line 112: Appendix A.2 should be Appendix A.4. [a] Language Conditioned Traffic Generation. CoRL 2023. [b] DriveSceneGen: Generating Diverse and Realistic Driving Scenarios from Scratch. RA-L, 2024 (arXiv 2309). [c] CaDRE: Controllable and Diverse Generation of Safety-Critical Driving Scenarios using Real-World Trajectories. arXiv 2403.

Questions

- Would amortized diffusion strategy have problems (such as different diffusion steps) on the boundary of the prediction window? - Is the predicted size for one specific agent not consistent during the prediction horizon?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Summary

The paper introduces SceneDiffuser, a novel scene-level diffusion model designed to enhance traffic simulation for autonomous vehicle (AV) development. It presents a unified framework that addresses scene initialization, involving the generation of initial traffic layouts, and scene rollout, which includes the closed-loop simulation of agent behaviors. SceneDiffuser leverages diffusion models to learn realistic and multimodal agent distributions, focusing on controllability, realism maintenance in simulations, and inference efficiency. The model introduces amortized diffusion for simulation, reducing computational costs and mitigating closed-loop errors. Additionally, it enhances controllability through generalized hard constraints and language-based constrained scene generation using large language models (LLMs). The paper demonstrates SceneDiffuser's effectiveness in the Waymo Open Sim Agents Challenge, achieving top performance among diffusion models.

Strengths

- **Innovative Approach:** SceneDiffuser's use of amortized diffusion for simulation rollout generation is a creative solution that significantly reduces the computational cost per step. - **Unified Framework:** The model's ability to handle both scene initialization and rollout in a unified framework is a notable strength, simplifying the simulation process. - **Controllability:** The introduction of generalized hard constraints and the use of LLMs for constraint-based scene generation offer a high degree of control over simulation scenarios. - **Performance:** Achieving top performance in the Waymo Open Sim Agents Challenge indicates that SceneDiffuser is effective in real-world applications. - **Scalability:** The model's performance improves with increased computational resources, showing that it can scale with available hardware.

Weaknesses

Despite mitigation efforts, the paper acknowledges that closed-loop errors remain a challenge, indicating room for further improvement. While the model performs well among diffusion models, it does not exceed the current state-of-the-art performance for other autoregressive models, suggesting a need for comparison and potential integration.

Questions

How does SceneDiffuser handle scenarios with a high number of agents or complex traffic situations not seen in the training data? How does the model ensure the diversity and representativeness of the generated traffic scenarios?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

- **Model Validity:** The paper does not explicitly model validity masks, relying instead on logged validity, which could be a limitation for scenarios not covered in the logs. - **SOTA Comparison:** While the model performs well among diffusion models, it does not exceed the current state-of-the-art performance for other autoregressive models, suggesting a need for comparison and potential integration.

Thanks for your reply

I appreciate the response and all generally makes sense. I'll maintain my score.

Response to Rebuttal

Thanks for the clarifications. - Please incorporate the details about metrics in the revision. High-level descriptions would be good in the main paper and you can put more details in the appendix, given the limited space. - The potentially inconsistent objects' size prediction is a little weird. However, this can probably be mitigated by certain post-process. - Concerning the more related works, I was curious if those provided WOMD results can be compared quantitatively. I read other reviewers' comments and authors' replies, and I also agree with their concerns such as SOTA comparisons and LLM's integration. Generally, the contributions of this work are technically solid. One last thing, it seems there will be no open-source code for this work. This is a pity. Official code can greatly promote the development of the community.

Thanks for the reply

I appreciate the authors' response and will maintain my score.