Summary
This paper proposes a motion-based method for video prediction. They design a new motion representation named motion graph that transforms patches of video frames into interconnected graph nodes. The proposed video prediction pipeline, empowered by the motion graph, exhibits substantial performance improvements and cost reductions. Experiments on UCF Sports, KITTI, and Cityscapes are conducted.
Weaknesses
- Motivation and Occlusion / Out-of-View cases. As the authors claimed in lines 33-37 and Figure 1, existing methods struggle to effectively handle occlusion cases. However, the paper does not provide a corresponding evaluation to demonstrate the effectiveness of the proposed motion graph in complex situations involving occlusion or out-of-view scenarios. This is a significant omission, as these challenges are prevalent in real-world video prediction tasks. Furthermore, based on my understanding of the work, the proposed motion graph may not adequately address these problems. Specifically, occluded and out-of-view object/background do not appear to be considered in the graph, as the connections and features captured in the motion graph are derived from the visible parts of the targets. This raises concerns about the method's ability to maintain accurate predictions when dealing with occlusions or objects moving out of the camera's field of view. The authors should provide a thorough discussion that includes both the theoretical principles and empirical evidence.
- Video prediction evaluation setting. While the proposed motion graph method shows promising results in predicting a few frames (up to 10), the paper lacks an evaluation of its performance in predicting a larger number of frames. Other video prediction works [A, B, C] have evaluated their methods on longer prediction horizons, providing a more comprehensive assessment of their models' robustness and accuracy.
[A] ExtDM: Distribution Extrapolation Diffusion Model for Video Prediction, CVPR24
[B] Efficient Video Prediction via Sparsely Conditioned Flow Matching, ICCV23
[C] MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation, NeurIPS22
- Comparison with MMVP. The proposed motion graph method appears to be quite similar to the motion matrix used in MMVP, which also provides a dense connection among frames. Although I appreciate the discussion provided in the related work, the authors do not sufficiently clarify the differences and advantages of the proposed method compared to MMVP. This lack of distinction makes it difficult to understand the unique contributions and potential improvements offered by the motion graph.
- Comparison with other motion-cues-guided video prediction methods. The paper lacks a comparison with other motion-cues-guided video prediction methods, such as MotionRNN, which also leverages a vector-based motion representation. The authors should explain how the feature warping mechanism in MotionRNN compares to the feature processing in the motion graph. What are the potential benefits or drawbacks of each approach? Are there specific scenarios where the motion graph's approach to feature handling provides a clear advantage?
- Lack of a component-wise ablation. The reviewer noticed that there is no component-wise ablation study for motion feature learning, upsampling, decoding, image wrapping, and interaction modules. It makes it difficult for the audience to identify which parts of the methodology are effective and contribute most to the overall performance. The authors should conduct a thorough ablation study that isolates and evaluates the impact of each component of their proposed method.
- Complexity of the proposed method and presentation. Actually, the proposed method is somehow non-trivial, making this work hard to follow. Accurate presentation can make the work easier to understand.