Summary
This paper tackles the problem of generating both the initial configuration and vehicle trajectories for driving scenarios given an input top-down map. It proposes a latent diffusion model that operates in the latent space of a learned autoencoder trained to decode vehicle boxes and trajectories. The model can be directly conditioned to encourage a subset of agents to adhere to user specifications. Experiments show that once trained on a large real-world driving dataset, the model can produce reasonable scenarios at multiple scales and allows for user control over a subset of agents.
Strengths
The problem is important for AV testing and training.
The proposed method simultaneously generates both an initial scene configuration and vehicle trajectories, instead of in two stages as in prior work.
The proposed latent diffusion model is novel for vehicle trajectories – all prior work I have seen is explicit trajectory diffusion and the design of the autoencoder for such a model is not trivial.
The model allows directly (optionally) conditioning on attributes of one or more agents which gives nice flexibility and allows generating variations of scenarios directly taken from e.g. a log or specified by a user.
Results of generating short 4 sec scenarios look reasonable. Experiments show the ability to generate scenarios of various sizes on the sizable Argoverse 2 dataset. Fig 4 is a neat demo showing that the model is picking up on the semantics of the relationships between vehicles.
The appendix is extensive and provides most of the necessary details to enable reproducibility.
Weaknesses
There are a couple claims that seem over-stated and should be toned down or clarified:
* Conditioning tokens are “more abstract features” (L208) that does not require “fully specifying box and trajectory for the agent” (L206) or “having observed such agents in the real world” (L43) – if I understand correctly, these tokens require a user to give the model the desired agent position, heading, length, width, speed, and sometimes curvature over time (L211), which is quite a low-level specification of the trajectory and likely will need to be taken from a log. Or do tokens describe the state only at a single $t=0$? If so, this should be made more clear, and I still would not say this is “abstract”.
* The model “generalizes to different geographical regions” (L9) – Table 3 shows seemingly limited ability to generalize across regions with models trained specifically on each region far outperforming other single-region models. The model trained on all regions does reasonably well on each, but this is not generalizing across regions. Moreover, it would be helpful to add a metric that indicates similarity between regions (e.g. MMD between ground truth scenes in A and B) to put the numbers in context. While this experiment is informative about performance across region, it may be a stretch to say it demonstrates generalization across regions.
Some limitations of the technical approach may make the proposed method less practically useful:
* The agent tokens are not hard constraints and in results like Fig 3 the specified agents move around between samples and don’t exactly meet user specifications. A natural solution for diffusion would be to use some kind of test-time guidance as in [37] and [Trace and Pace, Rempe et al., CVPR 2023].
* The 4 sec time horizon (2 sec past + 2 sec future) seems quite short and limits the possible scenarios it can capture.
* The conditioning requires a user to specify all attributes of an agent. But it seems with the masking strategy it should be possible to condition on partial information (e.g. specify the past trajectory but generate the future, or only specifying speed) which would greatly enhance the flexibility.
Some additional evaluation metrics would help to evaluate realism and diversity. For example, MMD on speeds, collision rate between vehicles, and a diversity metric (like average pairwise distance between multiple samples). Additionally, some key experiments are missing:
* The accuracy of the model’s controllability is never evaluated, which is the main claim of the paper. It would be good to report how closely generated agents follow the agent tokens specified by the user (e.g. L2 distance between positions, headings, speed, etc..). The 0% model in Table 2 kind of does this, but the reported MMD metric is not very intuitive.
* A comparison to a baseline to justify the advantages a diffusion model to generate both static states and dynamic trajectories jointly. E.g. a comparison to TrafficGen [6] or a simpler baseline that only generates static states and then uses a trajectory prediction or traffic simulation model to roll out motion, e.g. [BITS: Bi-level Imitation for Traffic Simulation, Xu et al., ICRA 2023]. Or evaluating if the trajectories generated by the diffusion model align with those from a SOTA trajectory forecasting model?
* It would be good to see an example of how these scenarios can be useful. For example, that training a driving policy on a dataset augmented with generated scenarios does better than just a real-world dataset.
* More qualitative results (ideally videos, but additional figures in the supplement would also be helpful).
Questions
Overall, I am borderline on this paper. While the latent diffusion formulation is novel and technically sound and the model nicely generates both starting positions and motion for the scenario, the primary contribution of “controllability” is not sufficiently evaluated, and additional evaluations/baselines would strengthen the paper. In the rebuttal, I would really like the authors to clarify my confusion on the agent tokens and ideally evaluate the controllability of the model by showing how closely it follows user specifications.
The method Sec 3 at times is rather high level and leaves some questions:
* Sec 3.1: why is the encoder not conditioned on the map like in the typical CVAE formulation?
* Sec 3.3: A lot of info about tokens are relegated to experiments in Sec 4.3.1 or the appendix but may be better to include in the methods. I’m still left wondering how exactly the cross attention works, e.g. how are the tokens associated with agents in the scene since there is no canonical agent IDs or ordering? Also, if tokens are specifying the properties of agent trajectories, why do multiple samples of the same scenario (e.g. Figs 1, 7, 8) exhibit varying trajectories for the specified agents?
* How are the number of agents in the scene determined by the diffusion model? Through the agent probability? Why is the global scene token represented as the masking probability and not directly the number of agents that are desired in the scene?
* The methods mentions 2 agent tokens are used, but then 3 are used in Fig 3 and N are used in Table 2, so it’s not clear what is the problem formulation and what’s used in practice.
Another other small suggestion: since explicit trajectory diffusion models are becoming more prevalent [37] [Trace and Pace, Rempe et al., CVPR 2023] [Stochastic trajectory prediction via motion indeterminancy diffusion, Gu et al., CVPR 2022] [Planning with Diffusion for Flexible Behavior Synthesis, Janner et al., ICML 2022], it would be good to discuss them in related work and compare to the proposed latent diffusion.
===================== After Rebuttal ======================
After considering other reviews and discussion with the authors and reviewers, I have decided to keep my rating leaning towards reject.
Given that the problem is relatively new and the latent diffusion approach is novel, I am willing to be lenient on the lack of a baseline comparison. Especially since TrafficGen (ICRA 2023) is concurrent work and SceneGen (CVPR 2021) did not release code. However, without a baseline, the experiments and metrics need to be comprehensive and convincing to show generated scenes are realistic and match user specifications; I don’t believe this is the case in the current draft. The rebuttal Table 1 adds some new map-based metrics, but important realism evaluations like collision rate, plausibility of trajectories (smoothness and comfort), and diversity are still missing. Moreover, the added evaluation of controllability in rebuttal Figure 3 is rather coarse (considered “correct” if within 2.2 m) and does not consider other user specifications like length, width, speed, and curvature.
I also think it would be a strong addition to show results on higher-level semantic conditioning of the model such as vehicle type and maneuver type (e.g. lane change, turning, or u-turn) to justify the claim that the model can handle “abstract” specifications rather than just partial explicit specifications.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
Limitations are sufficiently discussed in Sec 6.