DeMo: Decoupling Motion Forecasting into Directional Intentions and Dynamic States

Summary

In this paper, a decoupled decoding process integrating intention and future state querying is proposed for motion prediction task. Through hybrid design with attention and Mamba, the proposed DeMo framework achieved strong performance on AV2 and NuScenes benchmarks.

Strengths

1. A novel decoupled query design for intentions and future states. 2. A good trial in leveraging Mamba structure for efficient statewise decoding. 3. Solid performance on prediction benchmarks.

Weaknesses

1. An over claim for SOTA performance: For instance, the LOF [1] method showcases much better performance in AV2 compared with proposed DeMo. I think a thorough comparison and corrections should be conducted. 2. Heavy computational load for long-horizon state queries.

Questions

1. Whats is the computational cost such as FLOPs and training time of DeMo compared to other methods?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A.

Summary

The manuscript presents a novel decoupling method for motion forecasting tasks, where the directional intentions are predicted first and the dynamic states following the predicted direction are predicted accordingly. The proposed solution is easy-to-follow, the model size is small, and the experimental results are convincing, achieving the first place of the Argoverse 2 dataset challenge assumed by the reviewer.

Strengths

1. Simple model design with good performance 2. Extensive experiment and convincing results.

Weaknesses

1. Some unclear details about loss function and model design.

Questions

1. The loss function is simply a sum of all loss terms. Is it possible to have imbalanced losses, or in other words, one loss dominating the training? 2. Is there a Typo at Line 154? $Q_a$ or $Q_s$? 3. Between Lines 135-136, the authors said that $T_s$ and $T_f$ can differ. In this case, is the prediction conducted multiple times to fill the $T_f$ steps? If so, can the directional intention change in this process? If this is possible, can the model handle it?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

See above.

Summary

DeMo: Decoupling Motion Forecasting into Directional Intentions and Dynamic States introduces a state of the art model architecture for motion forecasting (predicting the future trajectories of road actors for the purpose of autonomous driving). The authors make two notable contributions. First they provide an alternative to the typical one-query one-trajectory paradigm of current high performing models. Instead they decompose queries into "mode queries" which attempt to capture directional intentions and "state queries" which attempt to capture the dynamic properties of a trajectory. The authors also replace the sequence processing portion of the encoder (typically cross attention layers in current models or RNNs in older models) with Mamba blocks. The authors include numerous ablations demonstrating that each of these contributions plays a substantive role in the performance of their model. The authors present results on two significant motion forecasting benchmarks.

Strengths

This is a very strong and well-presented manuscript. Some strengths include: ### Presentation - The copy is excellent. The paper is well written and largely quite clear. - Tables are excellent. - Visual examples (figures 3-5) are clear and compelling. ### Results The results are presented on two (of the three) most important motion forecasting benchmarks for self-driving. In both cases DeMo is clearly SOTA. They additionally present ensemble results. ### Analysis The authors conduct an extensive set of ablation experiments. They conduct the expected ablations (disabling various parts of their architecture). Additionally, they explore the impact of: layer types (RNN vs Uni MB vs Bi MB), the number of layers, the number of state queries, the auxiliary losses.

Weaknesses

This is already quite a solid manuscript, however I hope that the authors can selectively address a few of the weaknesses and questions below to arrive at an even better paper. ### Results While AV2 and NuScenes are both important benchmarks, WOMD (Waymo) is probably the most important (or co-most-important) benchmark in this space right now. It would have been nice to see results on WOMD. There are tools like https://github.com/vita-epfl/UniTraj now to make this easier. ### Analysis - While tables 4 and 5 clearly demonstrate that the query decomposition and auxilary losses contribute significantly to the model performance, there is little evidence to support the motivating intuitions. For example, the introduction of state queries and the associated auxiliary loss is supposed to produce better dynamics. Can we come up for a metric to measure that? Can we find ways to demonstrate that our decomposition works as we expect? Directional information is predominantly stored in mode queries while dynamics are encoded in state queries? - I would have liked to see the authors get an RNN based model to successfully converge. (The GRU result is disappointing). ### Reproducibility The authors promise to release code (which would entirely alleviate my concern here), however I do not feel that the model is reproducible from the manuscript alone. There is very little information about the auxilary losses. Additionally there is no supplementary diagram or table with an explicit architectural description (layer sizes, normalization, etc).

Questions

- Can you provide a precise formulation of the auxilary losses? - Can you report training time? - How does the model in ID3 (table 4) work? Are the decouple queries just concatenated and fed directly to the MLP decoder? - How does the model size vary across your ablations (table 4)? I.e. why should I ascribe the performance improvements to architectural choices and not just growing model capacity? - For inference speed can you report 99th percentile (or similar) rather than mean? The AV industry is primarily concerned with worst-case performance. - How does the multi-agent set up work? Do I re-encode the scene in each agent-centric frame? - Line 488 -- the text in this paragraph makes it sound like ZOH is a continuous->discrete transformation. I might be wrong here, but I typically think of it as a digital->analog transformation. Are we really doing something like the inverse of ZOH?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

No concerns.

Summary

The paper presents DeMo, a novel framework for motion forecasting in autonomous driving systems. DeMo decouples the motion forecasting task into two distinct components: mode queries for capturing directional intentions and state queries for modeling dynamic states over time. This separation allows DeMo to separately optimize for multi-modality and dynamic state evolution, leading to a more comprehensive representation of future trajectories. The framework employs a combination of Attention and Mamba techniques for global information aggregation and state sequence modeling. Extensive experiments on the Argoverse 2 and nuScenes benchmarks demonstrate that DeMo achieves state-of-the-art performance in motion forecasting.

Strengths

The overall idea behind DeMo is reasonable and technical soundness. Additionally, the experiments are comprehensive, demonstrating the results of DeMo across two different datasets, Argoverse 2 and nuScenes.

Weaknesses

There are two major weaknesses: 1. Technical Descriptions Lack Clarity: The technical explanations of the methods and algorithms used in DeMo are not sufficiently clear, making it challenging for readers to fully understand the proposed techniques and their implementations. 2. Unconvincing Contributions: The claimed contributions of the paper are not entirely convincing. For instance, the paper doesn’t compare DeMo to models that have superior performance on existing leaderboards, questioning the novelty of its improvements. Additionally, there are previous works (e.g. Motion Mamba by Zhang et.al.) that have already combined Mamba and Transformer techniques for time series motion data, which undermines the claim that DeMo introduces a novel approach.

Questions

1. In the first line of Eq. (2), what do the symbols {$\{ t_1, t_2, ...t_{T_s} \}$} represent, and how do $T_s$ and $T_f$ differ from or relate to each other? How are these time values obtained and utilized in the model? 2. For the motion model $Q_m$, what specific features are considered for different motion models? The paper does not clearly specify the features used in these motion models. 3. In Section 3.4, the loss function $L_{ts}$ is described as related to "intermediate features of time state." What exactly are these intermediate features, and what is their role in the model? This concept is not clearly defined in the paper. 4. Table 1 does not include baseline methods such as SEPT and SEPT++ that have better performance and earlier submission dates than DeMo. Similarly, Table 3 omits models like QCNet, which outperform DeMo. Why are these models not included in the comparisons? 5. The paper lacks detailed information about the model’s parameters and settings. Can you provide more specific details on the parameters used for DeMo?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

The technical descriptions lack clarity and the overall contribution is limited.