Motion Forecasting in Continuous Driving

Summary

This paper proposes a model for motion forecasting that models the continuous stream of world state over sequential timesteps. This is in contrast to most/all other works which do an indepedent prediction of the future for each timestep based on a fixed window of history, without context of previous model belief states or outputs from previous time steps. One stream of data is scene context, represented as embeddings, where context is fused with current accounting for transformation of the scene due to ego motion. The second stream is agent trajectories, where a history of n previous predicted trajectories for each agent is maintained explicitly in a memory bank, and fused into the current time steps prediction as additional signal. They do thorough ablations on Argoverse benchmark dataset, showing compelling evidence that the streaming architecture is beneficial. Furthermore, they exceed SOTA on most metrics in the benchmark. When I checked today (mid July), they are ranked 7th overall in the leaderboard - https://eval.ai/web/challenges/challenge-page/1719/leaderboard/4098

Strengths

This paper addresses an important problem often ignored in these sanitized datasets: that these models in real world application in an AV system run in a sequential streaming fashion, and (to my knowledge) all other works operate on each time frame independently (given a fixed window of historical state). The proposed solution itself is a good one, transforming old belief state into current reference frames in both embedding space and with spatial transformations where appropriate. At the same time, the complexity of the architecture is kept reasonably low, with each contribution justified. Results are very strong and well justified with thorough ablations.

Weaknesses

Minor, but I had some difficulty understanding some of the explanations: - Data reorganization / Figure 2: "we select several split points Ti along historical frame steps" --> how are they selected? what are "historical frame steps"? Can you more simply say this as, for example, "We split sequences evenly into sub-sequences"? - Around L158: you started using the term "modes" for the first time, where I think you should just say trajectories, or be more careful defining the tensors F_{mo} and Y_{mo} making clear their dimensionality. - You never clearly state the output representation of your model (which I assumed to be a weighted trajectory set). Without being more clear here, Section 3.3. Model training is sort of meaningless - for example, what is the classification loss classifying? (I'm 99% sure it's a softmax distribution over trajectory modes, but this is not said anywhere.)

Questions

See weaknesses

Rating

7

Confidence

4

Soundness

4

Presentation

2

Contribution

4

Limitations

None

Summary

This submission tackles the task of trajectory forecasting in autonomous driving. In particular, it proposes improvements in two aspects: 1/ Data reorganization: Current datasets are artificially split into non-overlapping segments. The authors propose to reorganize the data to have overlapping windows. 2/ Relaying context and predictions: since the sub-scenes now overlap, they can use context and predictions from the previous timesteps (corresponding to a different sub-scene) to improve predictions at the current timestep. Relaying context serves to provide more information about the surroundings (an agent could have been missed in the current frame for instance), and relaying the predictions would smooth out the predictions across time. They demonstrate the validity of their approach on Argoverse benchmarks, achieving state-of-the-art performances.

Strengths

- The proposed improvements are interesting, sound, and motivated. - The data reorganization, I believe, is a simple idea that would that should be pushed in the motion forecasting community to build a new standard that better leverage the data. It can also immediately be applied to other models and hopefully improve performances. - The relaying through time helps achieve strong results either with their architecture or on top of QCNet.

Weaknesses

- The results are not very conclusive: - Compared to QCNet, except for $ADE_1$/$FDE_1$ (which are not the main metrics usually considered for motion forecasting) the improvements are small. It would be useful to give more insight into why only these metrics improve. We also don't have the $ADE_1$/$FDE_1$ values of QCNet + RealMotion so we don't know if these findings are consistent across models. - The comparisons are not fair in terms of available information. Because of the propagation, RealMotion has access to more past information both in the past trajectories and agents, and the past map. A naïve way to integrate more past trajectories would be to just use longer past history. This should be a minimal baseline or ablation in the paper. If we also wanted to add agents seen in the past but not at the current timestep, it could be done with some masking or extrapolation scheme for missing frames. Remains the past map information, for which it's indeed more complicated to have a naïve baseline for a fair comparison, as simple aggregation could drastically raise the computation footprint so I can understand why the authors wouldn't want to test this baseline. - The loss functions, and in particular $\mathcal{L}_{refine}$ are not detailed enough. I can guess where they apply but it would be good if that was stated explicitly. - Figure 2 is unclear to me. Perhaps it could need more captions or a rework. - I think it would be easier to interpret the results if the architecture of RealMotion would be more clearly delivered, perhaps in a Figure (in Appendix?) that represent the full architecture detailing in particular how the different elements interact in the encoder and the decoder that are not given in Figure 3.

Questions

- Why do we see drastic improvement on ADE/FDE_1 but the other improvements are very tame? Does that imply most of the improvements are to the scoring function so that RealMotion gets a better top 1? If that's the case, what could be the mechanisms that explain this behavior? Otherwise, I think this paper rightfully questions some practices of motion forecasting in autonomous driving. Generalizing something along the lines of the proposed data reorganization would be very helpful for the community. However, I think the experiments on relaying might not be fair enough and need to be compared to simple baselines with more/better past history. We would also need a few more numbers to see if most of the findings and recommendations have a good chance at being consistent accross models.

Rating

6

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

- Limitations are discussed in the Appendix. - Broader societal impacts do not seem explicitly addressed despite what has been reported in the checklist. I'm not sure there are obvious uncontroversially negative societal impacts to be discussed though.

Summary

This paper proposes a framework named "RealMotion" highlighting the importance of continuous motion forecasting in autonomous driving. From the formulation perspective, RealMotion investigates predicting trajectories in a continuous sequence of timestamps instead of previous independent predictions. From the methodology perspective, RealMotion proposes the "relaying" modules that propagate historical scene and agent contexts into the current frame to enhance prediction. Finally, RealMotion achieves top performance on Argoverse benchmarks and proves the importance of continuous contexts.

Strengths

1. This paper investigates a foundational challenge in autonomous driving that is overlooked by previous people -- how to model the continuous contexts in motion forecasting. This problem is reasonable and is also something I would love to follow and work on. 2. The methods proposed by RealMotion are intuitive and reasonable. Relaying the historical context to current frames to enhance motion forecasting is indeed a novel contribution to autonomous driving. 3. The experiments prove the effectiveness of continuous contexts. In addition to the state-of-the-art performance, the comparison between RealMotion-I and RealMotion in Table 1 makes it a strong argument.

Weaknesses

1. The authors have missed some closely related works, such as streaming motion forecasting (e.g., Pang et al.) and other end-to-end autonomous driving (e.g., Gu et al.), where motion forecasting is continuous. For instance, Pang et al. also discover the setbacks of predicting trajectories on independent timestamps, formulate the "streaming" forecasting task, and propose a differentiable filter to refine the predicted trajectories. I acknowledge the newly designed modules by RealMotion, but I would expect the authors to have a better discussion of these relevant efforts. Pang et al. Streaming Motion Forecasting for Autonomous Driving. IROS 2023. Gu et al. ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queries. CVPR 2023. 2. RealMotion is also closely related to QCNet and based on its code. QCNet already has the interface of continuous motion forecasting and leverages a GRU to deal with the temporal contexts in queries. First of all, I would like the authors to clarify or discuss in the paper about the difference/improvement to QCNet more explicitly. In addition, I am also curious if it is possible to support that the "relaying" modules in RealMotion is better than QCNet's end-to-end mechanism. For instance, is it possible to plug the relaying module to QCNet and make a comparison? 3. I would like the authors to develop some metrics to analyze the improvement in the trajectories. More specifically, we wish to analyze "where does RealMotion improve the trajectory quality?" Since the main argument of this paper is "continuity," I am curious if the author can use a "fluctuation" measure like Pang et al. to prove that the trajectory smoothness is indeed improved. It is also encouraged to show some visual comparisons if space is allowed.

Questions

See weakness above.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed limitations. No additional limitations from my side.

Summary

The paper introduces an approach to iteratively process temporal scene context for the purposes of agent motion prediction. This differs from traditional approaches that ingest the whole context directly. The scene is processed in temporal chunks, where in each chunk contains agent-centric context consisting of other agents and map within 150m. Information is aggregated across chunks (i.e. over time) using two novel components: - "Scene context stream", where agent/map features from the past frame are transformed using Motion-aware Layer Normalization [40] and cross-attended by current frame features. - "Agent context stream", where agent predictions from the current frame are modified by attending predictions from past frames (stored in a memory bank and properly aligned). This 'streaming-type' method obtains very strong results on the Argoverse 1 and 2 datasets, and is shown to improve two different baseline model architectures based on [5] and [51] that consume the whole historical context in a single step. Some helpful ablations and latency studies of the proposed components is provided.

Strengths

- Strong idea to stream scene context into the model, shown to be effective even with short 2-3 'chunks' of context over 2-3 seconds. - Intuitive and well designed scene and agent context stream components that aggregate information across frames (or 'chunks'). The components are designed to model local information in agent-centric coordinates and aggregate it in a geometry-aware manner. - Strong overall results on Argoverse 1 and 2, and when comparing the baseline models the new technique is extending. - Mostly comprehensive related work section.

Weaknesses

- In the abstract/intro, I find some of the framing to be a bit misleading. Examples: - "7:This significantly simplifies the forecasting task, making the solutions unrealistic to use in practice". I do not find this claim to be well supported. The current methods still ingest a significant amount of relevant context. To me the contribution of the paper is to follow a streaming paradigm and design more domain-appropriate fusion mechanisms across time than the vanilla MLP or transformer ops that are used traditionally. - "31 - 36: existing methods all tackle an artificially narrowed and unrealistic motion forecasting task".... The methods still ingest 2-3 seconds of context, but perhaps not as effectively as the proposed work. That does not confirm this overly strong claim however. Also the paper only demonstrates good performance with up to 3s of context, that does not prove yet that much longer context is helpful or practical. - To a similar point, Fig 1 is also a bit misleading: current methods would directly ingest the available scene history in the current datasets, and fuse the information, even if the information fusion is not done with the same geometric and streaming intuitions. - In the method description, some details are unclear or could be explained better: - 83: "generate trajectory segments of identical length" Length can be misconstrued to be the length of the trajectories in meters, not that they are the same across chunks. This whole paragraph could be rewritten for clarity and a more helpful diagram could be shown -- illustrating what exactly is done for Argoverse 2 (one now needs to read all the way to Sec 4 to understand how this works). - "114: Additionally, we forecast a singular trajectory for auxiliary training purposes, focusing on the movement patterns of other agents. " Unclear what this means exactly. Do you produce only a single trajectory prediction in past steps? - "“Equal length into past and future” → confusing, assume fixed time interval in both directions. - The L_refine loss in Eq 5 is not explained, unclear what is being refined. My guess is that this is predictions in earlier chunks - What is exactly RealMotion without the context stream components? This is present both in the Ablation and in Appendix B.3. How do you fuse information across time then? - In the experimental results, some SOTA methods from the leaderboard seem to omit SEPT (ICLR 2023), which has higher numbers than RealMotion, even though it is cited. Why did you you omit it? - A couple more ablations or experiments would be helpful. - It seems that the method is similar to a filter, where the current chunk only attends to previous chunk (but not to the ones before). Is my reading of the paper correct? If so, an experiment where the method can attend longer context memory with multiple frames would be interesting. - It would also be interesting to try pretraining this method on more data examples (those can be produced by predicting shorter futures, then you can use more of the segments). And then fine-tune on 6s futures.

Questions

Main question is, do you agree with my reading that "the contribution of the paper is to follow a streaming paradigm and design more domain-appropriate fusion mechanisms across time than the vanilla MLP or transformer ops that are used traditionally. " As opposed to the strong claims in the current abstract / intro. Why did you omit SEPT from the leaderboard results (Table 1)? Please see my clarification or ablation/more experiment comments in the Weaknesses. I am particularly interested to understand why you did not try to attend to longer temporal context in the last chunk (if I read the paper correctly).

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

4

Limitations

The same RealMotion methods tried here could be tried on the Waymo Open Dataset which also has 9 second segments, but one could take the first 3 seconds as history and predict 6 seconds, as opposed to the standard 1s / 8s split that is mentioned in the limitations.