Summary
This paper mainly introduces a new dataset called GroundMoRe for motion-centric pixel-level language grounding in videos. It collects a large number of video clips with question-answer pairs and object masks and also proposes a new baseline method MoRA to handle the new task on the new dataset. Extensive statistics and experiments have shown the advantages of the dataset and the effectiveness of the proposed method.
Strengths
1.The paper is well-written and clearly presents the motivations of the proposed dataset as a new motion-centric video benchmark. And there are sufficient examples and figures given in the manuscript to illustrate the dataset characteristics.
2.I think the new task Motion-Grounded Video Reasoning proposed in this work is a more practical setting to benchmark the video multimodal models, since it considers the capability of temporal localization which is commonly neglected by previous methods.
3.The proposed baseline model is reasonable and shows effectiveness in addressing Motion-Grounded Video Reasoning on GroundMoRe.
Weaknesses
1.Although one of the most significant contributions claimed by the authors is to introduce implicit reasoning into the video question-answering and segmentation models, this idea has been previously explored by several works like VISA [1] and ViLLa [2]. It seems that there is no obvious difference in terms of the introduction of implicit textual inputs compared to these existing works, and this makes the contribution and novelty of this work not that impressive.
2.In terms of the Sequential question type in the proposed GroundMoRe dataset, I have some concerns on the effectiveness of it for truly reflecting the model's ability to reason about the temporal relations of motions. For example, as shown in Figure 1, the query input "Who dribbled the ball before he accelerates passing the man in pink shorts?" involves two consecutive motions, but there is only one motion of "dribbling the ball" in the video clip so the model can easily localize the first motion without looking at the second motion to find the correct answer, while this behavior cannot be regarded as temporal relation reasoning. I think similar problem will also occur in other examples mentioned in the manuscript like "the woman opened the refrigerator before taking out the milk". In my view, some samples should be explicitly constructed for this question type, where a same motion occurs twice but one of them is accompanied with another context motion, so the model has to rely on the temporal relation with the context motion to localize the answer.
3.To my understanding, the most interesting contribution of this work is to take the temporal grounding into consideration during the benchmarking process. However, it seems that the evaluation and investigation of temporal grounding capabilities are quite simple. For example, I don't see the authors discuss any metrics regarding the temporal grounding capabilities, such as the typically adopted temporal Intersection over Union (tIoU). Actually I think there should be a more comprehensive metric considering the spatio-temporal localization ability in this work, which can be derived from a simple extension to the existing vIoU metric proposed in Spatio-Temporal Video Grounding [3, 4]. But the current metrics discussed in this work are still focusing on the spatial segmentation.
4.According to the experimental results presented by Table 4, I feel kind of confused that for the Descriptive question type, the performance improvement is still remarkable when replacing the implicit questions with the referring expressions, i.e., the first row and second row for all methods in Table 4, and sometimes this improvement is even more obvious than other question types. Given that the Descriptive questions already convey a lot of description details, why could this phenomenon happen? Intuitively I think the difference between a descriptive question and a referring expression is quite minor, maybe there is just something like whether the concrete subject word is mentioned or not comparing these two kinds of textual inputs?
References
[1] VISA: Reasoning Video Object Segmentation via Large Language Models, Yan et al.
[2] ViLLa: Video Reasoning Segmentation with Large Language Model, Zheng et al.
[3] Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences, Zhang et al.
[4] Human-Centric Spatio-Temporal Video Grounding With Visual Transformers, Tang et al.