A Diffusion-Model of Joint Interactive Navigation

Summary

The paper deals with the problem of generating vehicle trajectories at the scene level conditioned on a map and some known observations (e.g. past or future states). They propose a diffusion model based on SceneTransformer that is trained with observation conditionings with random masking to reflect multiple desired downstream applications including prediction, goal conditioning, and imputation. Experiments show reasonable results on trajectory forecasting on both Argoverse and INTERACTION datasets, and demonstrate the flexibility of the model to several tasks including generating scenes based on high-level concepts (cut-in) and scenario editing.

Strengths

While no technical component of DJINN is particularly novel, the paper presents a nice comprehensive exploration of using diffusion for traffic scenario modeling. This includes all different types of masking/conditioning, classifier-free guidance, classifier guidance, and scene editing with SDEdit. Sec 3.2 is a succinct, yet informative, introduction to diffusion which will be good even for the unfamiliar reader in the driving community. In general, the paper is relatively self-contained, well-written, and easy to follow. The conditioning approach makes the model very flexible to many different kinds of observations that enable use in several important downstream applications. The paper goes beyond showing the usual motion forecasting applications (although it shows reasonable results on this too). The use of classifier guidance in Sec 6.3 to allow creating scenarios with more abstract specifications is a cool proof-of-concept that could be really useful for AV testing. Similarly for the scene editing application in Sec 6.4.

Weaknesses

In some cases, it looks like conditioning the model on input observations is not strong enough to be consistent with the observations. E.g. Fig 1 bottom row for Goal Conditioning, Up-Sampling, and Imputation the generated parts of trajectories in orange do not always align well with observations in blue, i.e. there are jumps or discontinuities. It may be necessary to use test-time guidance (as in CTG [52] and [Trace and Pace, Rempe et al., CVPR 2023]) to better enforce consistency. Also, it would be good to have an evaluation to quantify this inconsistency. E.g., for up-sampling and imputation the generated portions could be compared to ground truth, or for goal conditioning how close the vehicle gets to the goal. Some details of the masking procedure to allow this conditioning were also not clear. Does the model only output predictions for timesteps which there were no observations? Or does it generate a full trajectory and only the unmasked portion is visualized in the figures? If the latter, is the masked portion (i.e. the part of the trajectory given as input) supervised during training? The classifier-free guidance (CFG) formulation was also a bit confusing. The difference between $x_{obs}$ and $x_{cond}$ in Sec 5.1 was not apparent until later in Sec 6.2. Usually CFG operates by dropping the entire conditioning from the model in the right-hand term of Eqn 6, so I’m wondering why only some of the conditioning (e.g. the goal state in Sec 6.2) is arbitrarily dropped in the proposed formulation? In this example (and in others like motion forecasting), it seems the past conditioning should also be dropped. I also think on L275-276 it may be more accurate to say that the CFG controls the emphasis of the conditioning to the model instead of controlling the “spread of agents’ trajectories”: the reason the spread reduces may be because the goal position conditioning is emphasized so more samples are guided closer to this goal. The paper and supplement currently have only a few qualitative results, making it hard to comprehensively evaluate trajectory quality. Given the range of applications that the masking enables, it would be great to show several examples from each of these applications. Videos would be ideal, but at least trajectories should be colored with a gradient over time or something rather than a solid color over the whole trajectory. The observation distribution is detailed in Appendix B, but I’m curious about how this distribution was determined and how much it affects the final capabilities of the model? It seems balancing these probabilities would be very important to ensure good results, so I’m wondering does the model do significantly worse when trained on all tasks jointly than trained on one individually? One of the most useful applications of the proposed method could be in simulating scenarios for AV testing, but the current generated trajectories are quite short (at most 5 sec) and limits the kind of scenarios. I’m interested to see if the model can handle longer generated scenarios (~10 sec), and if the trajectories maintain realism using metrics like off-road and collision rate.

Questions

Overall, the paper is a nice exploration and demonstration of how diffusion can be applied to traffic scenario modeling, and I tend to think it would be a good contribution to the community. But in the rebuttal, it would be really good to see more qualitative results and for the authors to clear up some of my confusion on the masking and classifier-free guidance procedures. I had a few questions in Sec 4 that didn’t weight heavily in my rating: * Eqn 5: why is the loss on $x$ and not $x_0$? Is the entire trajectory supervised (observed + generated) or just the non-observed parts? * L186: why are the agent states called “latent”? Are they already embeddings of some kind? * It’s interesting that deterministic sampling is better quality than stochastic since this is not the case in other domains – any intuition here? ============== After Rebuttal =================== After the rebuttal and discussion, I have decided to raise my score to Accept. In the rebuttal, the authors addressed most of my concerns, clarified various technical points, and added more qualitative results that demonstrate the variety of applications.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations are discussed in Sec 7. It might be good to mention some limitations/failures specific to DJINN and not diffusion in general, e.g., mention that the model is not always realistically consistent with conditioning observations?

Summary

The paper proposes a generative model for producing synthetic traffic scenarios. The proposed method uses a diffusion model and learns to predict multiple agent trajectories jointly in a scene. The main contribution to previous methods is the increased flexibility of conditioning which can use different observation masks (corresponding to different use-cases), goal/non-goal, and other vehicle properties. The method is evaluated on two publicly available datasets and shows competitive performance to its deterministic counterparts.

Strengths

DJINN seems like a natural step forward for the multi-agent trajectory prediction task. In theory, the generative framework of diffusion models should allow for better sampling of the data distribution and produce traffic scenarios with larger variability that previous methods such as Scene Transformer [30] and TrafficSim [44]. This is important because datasets with traffic scenarios are highly imbalanced, containing very few safety critical events which are usually more interesting. The paper illustrates this flexibility through several interesting qualitative examples of test-time conditioning. Furthermore, the paper is well written, providing clear motivation, problem formulation, and background to the reader.

Weaknesses

Quantitative results in Tables 1,2 show comparable results to [30] in terms of forecasting performance but there is no quantitative experiment demonstrating the increased variability of generated traffic scenarios from the proposed approach. While this is shown qualitatively in sections 6.2, 6.3, and 6.4, and in my understanding [30] does not have the capability of test-time conditioning, I think a quantitative comparison (perhaps estimating variance over waypoint predictions) would still be useful. I would like to hear the authors' thoughts on this. The authors mention that the choice of the reference frame for the agent states is important. Perhaps an ablation study would provide more insight. What is the effect of using the observation masks during training besides defining different use-cases? How would the capability of the model to produce traffic scenarios with large variability be affected if only the "Predictive" mask was used? Please include a runtime comparison to baseline methods.

Questions

See weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The paper did not address the limitations and potential negative societal impact.

Summary

The paper proposes a diffusion model-based method for generating joint or conditioned predictions of traffic agents. The proposed method combines the EDM diffusion and SceneTransformer model architecture and provides different guidance for conditional predictions. The experiment results indicate comparable results of the proposed results with SOTA.

Strengths

The paper is well-written and the proposed approach gives readers very useful information about how the two SOTA methods (EDM and SceneTransformer) could be combined and its potential performance.

Weaknesses

In my understanding, the novelty of the paper is mostly on the combination of existing methods, which is OK but not outstanding. From the experiment results, the proposed method doesn't show clear improvements compared with other methods, which is a bit surprising given the combination of two powerful approaches. It would be great if the author could give more analysis and explanations on why this is the case.

Questions

It would be great if the author could give more analysis and explanations on why the performance of the proposed method doesn't show clear improvements.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

2 fair

Limitations

As mentioned in the paper, the long inference time limits the method to be more useful in practice.

Summary

DJINN (Diffusion-based Joint Interaction Network) is an innovative generative model that creates, edits, and forecasts multi-vehicle traffic scenarios in a stochastic manner. Leveraging diffusion models, DJINN also addresses the challenge of generating traffic scenes conditioned on a flexible configuration for the observation space. This flexibility of the model makes it well-suited for generating safety-critical events and out-of-distribution scenarios, which are extremely relevant for evaluating the performance of autonomous vehicles. Furthermore, by jointly diffusing the trajectories of all agents and conditioning them to customizable observation windows, the authors provide a fresh and innovative way of forecasting multi-vehicle traffic scenarios. The authors validate the efficacy of DJINN by benchmarking against state-of-the-art trajectory forecasting methods using the popular Argoverse and INTERACTION datasets. The results show that DJINN outperforms existing models like Scene Transformer in joint motion forecasting metrics, demonstrating its promise for contributing significantly to the development and safety testing of autonomous vehicles. Moreover, DJINN’s flexibility is demonstrated through its successful generation of goal-directed samples, examples of cut-in driving behaviors, and editing replay logs.

Strengths

Originality: The paper represents a highly original contribution to the field of autonomous vehicle simulation. The use of a diffusion model for generating joint traffic scenarios is innovative, and appears to be a novel application of this technique. The authors move beyond the conventional deterministic sets of trajectory forecasts, proposing a generative model to forecast joint future motion. DJINN's ability to draw traffic scenarios from a variety of conditional distributions further attests to its innovative design. The model also stands out in its ability to provide flexibility in terms of test-time diffusion guidance, offering a new perspective on traffic scenario simulation. Quality: The overall quality of the research is high. The paper presents a sound background and solid related works section, creating a strong foundation for their argument. The authors have clearly demonstrated their understanding of the topic, using relevant and current research. Their methodology is rigorous, and the experiments performed on the Argoverse and INTERACTION datasets are thorough and well-executed. The results obtained are credible and validate the authors' claims about DJINN's performance and flexibility. Clarity: The paper is well-written and clear. It efficiently communicates the problems associated with the simulation of autonomous vehicle systems and convincingly presents DJINN as a potential solution. The structure of the paper is logical, and the flow of ideas is coherent, making it easy for readers to follow the authors' argument. Significance: The significance of this paper is recognizable. Simulating diverse, safety-critical and stochastic traffic scenarios is of paramount importance in the field of autonomous vehicles. Its flexibility in controlling the conditioning of traffic scenes, coupled with its excellent performance on trajectory forecasting, makes it a valuable tool for researchers and developers in the field. Given the increasing interest in autonomous vehicles, this work is likely to have a high impact on both academia and industry.

Weaknesses

While the paper represents a significant contribution to the field, there are a few areas where the authors could improve upon in the future iterations of their work: - Detailed Analysis of Limitations: While the conclusion does mention certain limitations, the paper would benefit from a more detailed and separate section dedicated to the limitations of the proposed method. A thorough examination of the constraints and failure cases of DJINN could offer readers a more balanced perspective and help to guide future research efforts. - Future Directions: The authors might also consider providing clearer guidance on the future direction of the work. While DJINN has shown promise in its current state, discussing potential extensions and enhancements would be valuable. For instance, how might DJINN be adapted to handle more complex scenarios, or different action spaces and robotic domain? - Qualitative Analysis of Failure Cases: Alongside quantitative performance metrics, a qualitative analysis of failure cases would provide more comprehensive insights into the model's performance. Discussing specific scenarios where DJINN fails to generate accurate traffic scenes would offer readers a better understanding of its potential weaknesses and areas for improvement. - More Sample Traffic Scenes: The paper could also benefit from a larger set of sample traffic scenes, either included in an appendix or made available online. These examples would provide a more tangible sense of the model's capabilities. Video demonstrations would be particularly effective, allowing readers to visually appreciate the flexibility and realism of the scenes generated by DJINN. - Ablation Studies: The paper could also greatly benefit from conducting ablation studies to understand the influence of different components or variations of DJINN's model architecture on its overall performance. A detailed ablation analysis can help to pinpoint which elements of the model contribute most to its successful generation of traffic scenarios. This could include investigations into the impact of the diffusion process, the role of different conditional distributions, or the effect of different types of state observations. Understanding these aspects in more detail would not only provide further insights into DJINN's performance and functioning, but also guide future optimization and refinement of the model. Minor comments: Unlinked reference at line 88.

Questions

- How might DJINN be adapted to handle more complex scenarios, higher action spaces or other robotic domain? - Given that the inference time of DJINN it relatively slow, due to its diffusion structure and iterative estimation of the score function. Do you have any quantitative results on the tradeoff between computing time, performance and variation of network? (number of layer, depth, number of diffusion steps, etc) - How suitable would the model, as it is, be for a model of predictive control setting and real life deployment?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

While the authors acknowledge some limitations of DJINN in the conclusion, particularly the relatively slow inference time, a more comprehensive and explicit discussion of these drawbacks would enhance the overall balance and depth of the paper. Moreover, this section could provide a more explicit roadmap for future research directions. While the paper showcases DJINN's potential, there is room to explore its applicability in other contexts, or the implications of further improving its inference speed. Articulating these directions can inspire and guide subsequent work in this line of research.

Response to rebuttal

I thank the authors for their detailed response to my questions and concerns. Most of my issues have been resolved or clarified, and I intend to keep my rating of accept. However, for additional qualitative results I do not see an attached pdf to the individual response or the top-level response. Is it still possible to upload this?