Summary
The paper proposes a simple yet effective framework for video localization and question answering using pretrained image-language models. The key idea is to introduce a LOCALIZER module which localizes the keyframes in a video in order to ignore irrelevant frames and better answer the question. The LOCALIZER module is realized with a separate image-language model and the module can be further improved by fine-tuning on the pseudo labels generated by the QA results from the ANSWER module. The proposed framework, SEVILA, achieves state-of-the-art VQA results on multiple challenging benchmarks under both fine-tuned and zero-shot setups.
Strengths
1. The proposed framework is simple yet effective in extending image-language models to tackle video-related tasks such as video question answering and moment localization. The idea of using the LOCALIZER module to detect keyframes in a long video is technically sound and the results are convincing.
2. The paper provides extensive ablation study to analyze different components of the framework under different setups. E.g., the impact of selecting keyframes for VQA, the impact of self-refinement for the LOCALIZER module, and how the LOCALIZER compared with some existing keyframe localization methods. While there exists many keyframe localization methods, it's interesting to see the language-model-based method used in this paper gets comparable or even better results in the ablation study.
3. The paper is well organized, clearly written and the description of technical details is clear and easy to follow.
Weaknesses
1. The technical contribution of the work is relatively weak. The idea of selecting keyframes in a video for VQA is straightforward and widely explored in prior work. The proposed self-refinement strategy is also a standard semi-supervised learning approach and there's no new training strategies / techniques proposed for the fine-tuning of image-language models.
2. While I understand the main goal of this work is to utilize image-language models for VQA, the limitation of image-based models for video-related tasks should not be ignored. First, image-based models may not be sufficient to accurately localize the keyframes in a long video. Second, temporal modeling is also missing in the ANSWERER module, which makes the model less capable of modeling temporal dependencies compared with video-based models. These limitations may not be significant in the current VQA benchmarks since the questions are less sensitive to temporal dependencies and reasoning.
Questions
1. Is it necessary to use two different Q-former for the LOCALIZER and ANSWERER? Since both models are image based, what if we share the same Q-former for both modules?
2. What if we make the self-refinement an iterative process? In other words, we refine the ANSWERER module after the self-refinement of LOCALIZER, and vice versa. It's interesting to see when the performance would saturate with such interactive refinement process.
A minor suggestion: I don't think Figure 4 is a good qualitative example because (1) the action of the ladies is barely visible in the small figure, let alone the intention of the action; (2) the question can be answered by only checking the question itself without knowing the visual content, since "staring out" and "hands above their eyes" are explicated mentioned.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The authors have discussed the limitations of the work and its potential social impact.