MGF: Mixed Gaussian Flow for Diverse Trajectory Prediction

Summary

This paper proposes to solve the trajectory prediction task by normalizing flow based model with mixture Gaussian assumption. The trajectories are clustered to multiple Gaussian distributions in the pre-processing stage over training data. During inference, priors are sampled from the mixture Gaussian distribution as the guidance of trajectory prediction. Moreover, new metrics are introduced to evaluate the diversity of multiple trajectory generation.

Strengths

1. The authors analyze the limitations of current metrics ADE and FDE, which only focus on trajectory level prediction accuracy. 2. The idea of controllability for trajectory prediction based on learned mixture Gaussian distribution is interesting, although I do not think it is an appropriate task to demonstrate the controllability (see details in Questions)

Weaknesses

1. The mixture Gaussian assumption is not accurate. Human trajectories are always affected by the interaction with other pedestrians and the environment. I agree that the possibility of trajectory conditional on i) historical trajectory ii) trajectory of other pedestrians iii) environment (e.g. maps) could be a mixture Gaussian distribution. However, all trajectories follow mixture Gaussian distribution without any conditions may not be an good assumption. Because the scenes in training and testing set are different. The learned distribution only fit on the training set with specific scenes, which cannot be generealized to different scenes. 2. The proposed diversity metrics is not reasonable. The diversity in multiple trajectory generation is necessary only on the premise that the generated trajectories make sense. The definition of APD and FPD only consider how different amont the generated trajectories but ignore whether the trajectories are reasonable or not. For example, 20 future trajectories are manually set to equally cover all directions from time step t_1, which will have very good score on APD and FPD but we cannot say these future trajectories are good predictions. 3. This paper ignores lots of recent works in SOTA comparison. E.g. Mangalam, Karttikeya, et al. "From goals, waypoints & paths to long term human trajectory forecasting." ICCV. 2021. Weibo Mao, et al.. Leapfrog diffusion model for stochastic trajectory prediction. CVPR, 2023. Jiangbei Yue, et al. Human trajectory prediction via neural social physics. ECCV, 2022 Yonghao Dong, et al. Sparse instance conditioned multimodal trajectory prediction. ICCV, 2023. ... Although authors mentioned that for some of the recent works, the processed data is slightly different. It would be more convincible to comapre to them with same training/testing data instead of ignoring these works as these works represent the most recent progress compared to original benchmarks such as socialGAN.

Questions

Please see the questions in weaknesses. Besides, here are some suggestions. 1. The readability of notations and equations can be improved. For example, the paper does not include the scene map modeling, then map M (line 111) is useless. If O_{t0:t1} (line 119) only contains map information, then O can also be excluded from equations. 2. This task may not be the appropriate task for the proposed method. Although the task of trajectory prediction contains multiple trajectory output which usually involves generative models, this task is not a pure generation problem. In this task, the accuracy is still with highest priority. For example, a self-driving vehicle need to predict a pedestrian's (or another vehicle's) future trajectory to avoid collision, where only the correct trajectory is required. Since motion intentions of agents cannot be determined by their historical positions (as described in introduction), multiple predictions are necessary to cover the most possible futures instead of multiple predictions as diverse as possible (or controllable predictions). I believe it would be more appropriate to evaluate the proposed method in the tasks such as motion simulation in synthetic data, where diversity and controllability motion trajectory is useful to make the agents look real.

Rating

3

Confidence

4

Soundness

3

Presentation

2

Contribution

1

Limitations

Authors discussed the limitations in the appendix.

Summary

Due to the asymmetric and multi-modal nature of future trajectories, the authors point out that the standard Gaussian prior with a neural network-based transformation is insufficient for probabilistic trajectory prediction. They propose Mixed Gaussian Flow (MGF), a method that uses the mixed Gaussian prior in the normalizing flow model. MGF allows diverse and controllable generation by leveraging the prior distribution. A minimum-of-M loss is designed for the inverse process to increase the predicting precision. Substantial experimental results on well-known datasets like ETH/UCY and SDD verify the effectiveness of MGF.

Strengths

S1. The motivation is strong and the method is intuitive. S2. MGF allows controllable generation by manipulating the prior distribution. It also secures under-represented future trajectory patterns. S3. The experimental results are solid and convincing.

Weaknesses

W1. This paper challenges the standard normal prior assumption in the normalizing flow model, but lacks in theoretical support. Also see Q6.

Questions

Q1. The prior distribution is obtained by clustering trajectories in the training set. Will it suffer from a distribution shift in the testing set? Specifically, in ETH/UCY dataset, the testing set contains scenes that do not appear in the training set. Can MGF deal with this issue? If not, can you figure out any possible solution? Q2. The asymptotic behavior of ADE/FDE is well-studied [Ref.1]. Is it possible to theoretically or empirically analyze the asymptotic behavior of APD/FPD for a sufficiently large $M$? Is the value of APD/FPD sensitive to $M$? [Ref.1] Analyzing the Variety Loss in the Context of Probabilistic Trajectory Prediction. In ICCV, 2019. Q3. Some other papers have proposed metrics for measuring the diversity of trajectories, like the average maximum eigenvalue [Ref.2] and the energy score [Ref.3]. Can you discuss the connection and difference of APD/FPD with these metrics? Is it possible to evaluate the trajectory diversity of different methods using these metrics? [Ref.2] Social-implicit: Rethinking trajectory prediction evaluation and the effectiveness of implicit maximum likelihood estimation. In ECCV, 2022. [Ref.3] Evaluation of Trajectory Distribution Predictions with Energy Score. In ICML, 2024. Q4. In Section 4.4, how do you edit the directions of the cluster means? Q5. Can you provide some empirical evidence for the claim of "limited and imbalanced data" in Line 30 of Section 1, e.g., the weights of each Gaussian component of the prior distribution? Q6. Is it possible to find some theoretical support for the claim that "deriving a desired complex distribution from a simple and symmetric prior distribution is challenging, especially with limited and imbalanced data" in Line 29-30 of Section 1? Q7. In Line 230-232, the authors say "Instead, it is hard to back-propagate the gradient through the sampling process of a Gaussian mixture with the reparametrization applied." I think the gumbel-softmax trick can be used for selecting components while allowing back-propagation. What do you mean by saying "it is hard to back-propagate"? Comment. C1. The authors clearly describe the subtle differences in evaluation in existing methods. It really saves my time and helps researchers in trajectory prediction. C2. Appendix B contains some important facts. Please summarize these discoveries in the main text. **Update:** I have read the authors' response. They have addressed most of my concerns. I decide to raise my rating to 7. I hope that the authors can better organize the paper so that the presentation is clearer and it is easier for the readers to grasp the paper's significance.

Rating

7

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

N/A

Summary

The authors proposed a new normalizing flow-based human trajectory prediction method called Mixed Gaussian Flow (MGF), which promotes diversity and controllability of prediction. The model uses a mixture of Gaussian model as the initial distribution to transform, rather than single-modal standard Gaussian distribution. The paper proposed using clustering of training data to mine modes of movements and use these modes to fit the initial sets of Gaussians for the mixture. The authors also claimed they proposed new metric measure diversity (APD, FPD). The authors have shown MGF can exceed SOTA performance on ETH-UCY and SDD datasets during their quantitative analysis, and showcase by controlling the mixture of Gaussian, the types of trajectory to generate can be controlled.

Strengths

- The paper is well motivated, and by adopting Mixture of Gaussian, the model was able to achieve diversity and controllability - The paper has shown normalizing flow-based HTP model can achieve SOTA performance with good design - The model has good analysis (4.3, 4.4) showcasing the diversity and controllability of the model output.

Weaknesses

- Line 112, 113 using $t_0, t_1, t_2$ might not be the best choice, the reviewer suggest maybe consider $t_0,t_1, \dots ,t_{c}, \dots, t_{T}$ - Suggest tidy up the variable representation of this section, for example using $X$ for observation sequence and $Y$ for future sequence - Line 159 160, about normalizing flow overfitting, it would be great if the author could provide a reference to this claim or if this claim is by the authors themselves (also 129-130) - Line 218-219, NLL loss is in sub-Gaussian, but how to decide the mixing coefficient (the reviewer is aware line 199 says it is decided from training data but specifically how?) - Line 332, typo what is AED for APD or ADE? - The ablation shows the ADE and FDE performance had a big increase with the prediction clustering, without prediction clustering MGF will have performance similar to AgentFormer and GroupNet and will not achieve SOTA, so does this means prediciton clustering/interntion clustering is decisive in accuracy performance of the model? - The authors claimed they proposed APD metric, but it seems it is already in used in previous literatures under the same name and same formulation @inproceedings{yuan2020dlow, title={Dlow: Diversifying latent flows for diverse human motion prediction}, author={Yuan, Ye and Kitani, Kris}, booktitle={Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part IX 16}, pages={346--364}, year={2020}, organization={Springer} }

Questions

- Abalation study was on ADE FDE only, but what is the effect on the diveristy? It would be great if the author could showcase this for the abalation, as diversity is the main contribution of the paper. - If taking out mixture of gaussian and only use predictive clustering would the model achieve same level of ADE/FDE performance? - Besides, this quesiton it would be great if the author could address the concerns in weakness section. - The reviewer will consider increase the score if the authors could address and respond to the concerns mentioned above.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The reviewer appreciate the author mention in the supplementary about the limitation, where the model does not utilzed map information and predicitons will contain collision among agent. Although per guideline, the reviewer shall not punish this limitation, however, it is worth pointing out diversity without context might not be a sound logic. As solely promoting diversity might making the prediction output actually unrealistic considering the environmental and social context of the prediciton output. Although the author mentioned potential way to incorporating map information in the main paper, such proposal is more on the implicit side (as latent embeddings), which might not be able to address environmental collision issue.