MomentDiff: Generative Video Moment Retrieval from Random to Real

Summary

This paper tackles the video moment retrieval task from the generative perspective and proposes a diffusion-based localization model, named MomentDiff. It could sample random temporal segments as initial guesses and iteratively refine them to generate an accurate temporal boundary. Moreover, this paper proposes two“anti-bias” datasets with location distribution shifts to evaluate the influence of location biases. Experiments on three public datasets validate the effectiveness of the proposed approach.

Strengths

1. This paper addresses the cross-modal moment retrieval task using a diffusion-based model, which is interesting. 2. This paper builds two datasets with location distribution shifts, which is valuable for this research community. 3. Experiments on three datasets: Charades-STA, QVHighlights, and TACos, demonstrating the effectiveness of the proposed approach MomentDiff.

Weaknesses

1. Despite the widespread use of datasets like TAcos, Charades-STA, and ActivityNet Captions, this paper chose not to conduct experiments using ActivityNet Captions. 2. Previous studies [1][2] employed CharadesCD and ActivityNet-CD to examine the influence of location biases. Nevertheless, this paper made the decision not to directly employ these datasets. Why? [1] Towards Debiasing Temporal Sentence Grounding in Video [2] A Closer Look at Temporal Sentence Grounding in Videos: Datasets and Metrics 3. To provide comprehensive evaluation, comparisons with other supervised, weakly supervised, and zero-shot moment retrieval methods are crucial. Examples of such methods include [3] DORi: Discovering Object Relationships for Moment Localization of a Natural Language Query in a Video, [4] Structured Multi-Level Interaction Network for Video Moment Localization via Language Query, [5] Multi-Modal Relational Graph for Cross-Modal Video Moment Retrieval, and [6] Language-free Training for Zero-shot Video Grounding.

Questions

1. Could you please explain how to obtain the values of $Q_{\hat{v}}$, $K_{\hat{v}}$, and $V_{\hat{v}}$ mentioned on page 4, line 146? 2. Regarding the use of span embedding as the query in Intensity-aware attention instead of combining it with the textual query, could you please elaborate on the reasoning behind this decision? Additionally, it would be helpful to know if any experiments were conducted to validate this choice and provide justification. 3. This paper aims to incorporate audio features and integrate multi-modal video information. Could you please explain the methodology used to integrate the multi-modal video information? Furthermore, it is important to elaborate on how the paper demonstrates that the performance improvements are not solely a result of introducing audio information.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

no

Summary

To deal with the problem of temporal location bias, the authors propose a diffusion-based video moment retrieval framework, MomentDiff. They introduce the diffusion process into temporal localization from a generative perspective, and gradually generate real span coordinates from coarse to fine. Compared to learnable queries, the random noise input to the model reduces the dependence on the location information of the dataset. Therefore, MomentDiff achieves better results on two "anti-bias" datasets with changing location distributions. Besides, MomentDiff consistently outperforms state-of-the-art methods on three public benchmarks.

Strengths

a) This work proposes a novel and effective diffusion framework on the video moment retrieval task and alleviates the important location bias problem. The paper is also well motivated and well written. b) To demonstrate the robustness of the model, they propose two anti-bias datasets, which seem to be one of the main contributions of the paper. Promising experimental results. c) The paper presents promising experimental results. Authors will provide code and datasets.

Weaknesses

While I don't see obvious weaknesses, there are a few minor suggestions, and additional questions that the author needs to answer carefully: a) Please revise the notation and typos of the paper. For example, \epsilon in Eq 6 is not clearly defined above, only \epsilon_m. b) Please re-check the paper and correct errors on formatting, grammar, etc. For example, L154. The Eq (1) uses “Snj”, “Spj”, but L154 writes “Spj”, “Sni”. c) Some of the figures in Fig. 1 and Fig. 3 are so small that they are difficult to see even when zoomed in.

Questions

In image generation tasks, images generated by diffusion models are often diverse. With different random noise inputs, are the span coordinates generated by the MomentDiff model for the same video-text pair quite different, and is the model performance stable? If the result is relatively stable, what is the reason?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

The paper argues that if the user enters words that violate the law, the model may have a potential negative impact. I think sensitive word filtering technology can effectively solve this problem.

Summary

This paper first tackles video moment retrieval from a generative perspective, and proposes a novel framework called MomentDiff based on recently proposed technique Diffusion Models. MomentDiff can generate correct results from random spans, which can resist the temporal location biases. The experiments on three public datasets and two anti-bias datasets proposed by the authors demonstrate the effectiveness of MomentDiff.

Strengths

1. The generative perspective for video moment retrieval is novel. 2. The designed MomentDiff is effective for the location bias problem and easy to reproduce. 3. Experimental results on three public datasets and two anti-bias datasets demonstrate the effectiveness of the proposed method.

Weaknesses

1. From a generative perspective, traditional generative models like GANs can also be applied to the video moment retrieval task. Do the authors believe that these methods can be used, and if so, what is the difference between GAN and Diffusion models in this task? If not, please provide a reason. 2. It is unclear how other methods have solved the problem of location biases. It would be helpful for the authors to compare and contrast the advantages of the proposed method with existing solutions. 3. The total loss function is missing. Please clarify whether both loss functions L_{sim} and L_{vmr} are weighted 1.

Questions

See the Weaknesses.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The author points out that too many iterations will slow down the inference speed, and the solution is to reduce the number of iterations and make a trade-off between performance and speed. It is recommended that the author give the iteration round parameters on all datasets to show better trade-off.

Summary

This paper proposes a novel generative approach, MomentDiff, to address the Video Moment Retrieval (VMR) task. It replaces traditional dense or learnable proposals with random spans and a diffusion-based denoiser to refine predictions, mimicking the human process of identifying key video moments. This reduces the impact of temporal location biases and improves the system's generalizability. The authors also introduce two "anti-bias" datasets, Charades-STA-Len and Charades-STA-Mom, for evaluation. Experiment results showed that MomentDiff outperforms existing methods in efficiency and transferability.

Strengths

1. The proposed method creatively combines pre-trained video and text backbones for feature extraction, a similarity-aware condition generator, and a video moment denoiser. This composite approach takes existing tools and blends them in a unique way. The inclusion of audio data as a feature, alongside visual and textual data, also represents an innovative approach to video moment retrieval. 2. The paper showcases a high-quality approach by incorporating various feature extractors, utilizing a multilayer transformer for multimodal interaction, and deploying a similarity-aware fusion embedding. The fact that the paper also discusses the limitations of the proposed method speaks to its quality and rigor. 3. The proposed methodology is outlined clearly and in a structured manner. Each part of the system, from feature extraction to the inference process, is explained with sufficient detail. However, some areas could benefit from additional explanation (e.g., the impact of the quality of fusion embeddings on the denoising process), which could further enhance clarity. 4. The paper tackles the important problem of video moment retrieval, which has broad implications in fields like media indexing, recommendation systems, and video summarization. The solution proposed in the paper, especially with the inclusion of audio features, can be significant in improving the efficiency and effectiveness of video moment retrieval tasks. By outlining its method clearly and discussing potential limitations, the paper contributes to further research and improvement in the field.

Weaknesses

1. The proposed method relies heavily on the effectiveness of the chosen feature extractors. Although they have tested multiple feature extraction models, the paper does not discuss the impact of these choices on the final results in detail. Additionally, the models chosen for feature extraction could potentially limit the generalizability of the approach to datasets significantly different from those on which the models were trained. 2. The paper does not provide a clear comparison with existing methods in terms of computational resources. This makes it hard to gauge the improvement the proposed method offers over current techniques. 3. The paper mentions multiple hyperparameters but does not discuss how they are selected or tuned. This could impact the replicability and robustness of the model across different datasets.

Questions

1. Clarification on Visual and Textual Representations: It would be helpful if the authors could elaborate on why they chose the specific visual and textual extractors, like VGG, C3D, CLIP, Glove, etc. Are there specific reasons these were chosen over other potential extractors? 2. Elaboration on Span Generation Process: In the span generation process, it is mentioned that for the same video, the correct video segments corresponding to different text queries are very different. Could you elaborate more on this? Is there a way to address this challenge? 3. Justification for Hyperparameter Choices: Could the authors provide further clarification on the selection of the hyperparameters used in the model? How were these optimized, and what was the impact on model performance? 4. Scalability of the Model: Could the authors discuss how this model scales with larger, more complex datasets? Can the method efficiently handle real-world scenarios with high volumes of data, and if so, are there any limitations or performance degradation? 5. Use of Pre-Trained Models: What are the implications of using several pre-trained models? How does it affect the generalizability of the proposed method across diverse datasets, especially ones that differ significantly from the datasets these pre-trained models were trained on? 6. Computational Resources: Could the authors provide details about the computational resources required for the model to run both in the training and inference stages? This is crucial for evaluating the practicality of the proposed model. 7. No motivation is provided in Similarity-aware Condition Generator, i.e., why specific modality features are selected as Query, Key, and Values? Why not any other combination?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

1. Authors have provided limited limitations. 2. Code is not provided in supplementary that can help with in more detailed understanding.

Summary

This paper first proposes a diffusion model for video moment retrieval to overcome proposal-based moment retrieval and distribution-specific methods.

Strengths

[+] First work bridging generative frameworks into deterministic task as video moment retrieval. [+] Illustrative presentation [+] writing is easy to follow

Weaknesses

[Motivation] [1] There are many solutions (i.e. 2DTAN, VSLNet) without relying on moment proposals. Fully-supervised VMR does not concern about proposal generation, as it is frame-level supervision is available, where the previous works already design the regression-based (i.e., Attention Based Localization Regression) methods. [2] As this paper proposes a new framework (diffusion framework) for the VMR, what the previous frameworks of VMR are relying on the distribution-specific proposals? This paper refers to the 2DTAN, the framework of 2D map with respect to start-time and end-time can allow all the possible moments, where the framework is not relying on the distribution-specific bias. I think the authors may understand the applying the masking in the map as resulting distribution-specific proposals, which is more related to the heuristic filtering by empirical experiments, not a bias problem. [3] Why the diffusion (generative framework) more can be better than the deterministic models? In fact, the authors' proposed method is presented to overcome the proposal-learnable methods, however, current VMR method is not relying on proposals. (rather, the weakly-supervised method is dependent on proposals). Furthermore, location bias problem is sourced from dataset distributions, proposed diffusion framework does not correlate to mitigating bias. [Method] [1] Preliminary section is recommended in the paper or appendix about the forward-backward process of diffusion framework (e.g., denoising diffusion probabilistic model, conditional diffusion, sampling) to enhance the readability. [2] x_{0} is a 2-dimensional vector of center point and width. Does this paper truly add Gaussian noise on to the 2 values and denoise them? [Experiment] [1] I can not trust the performances in Table. Is there any experimental qualitative evidence why denoising frameworks can guarantee more performance than previous work?

Questions

I want to get answers about the weaknesses above in the rebuttal.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

See in the weakness.

Response to authors

The authors' response addresses some of my concerns. I will adjust my score to "borderline accept."