Video Question Answering (VideoQA) has emerged as a vital tool to evaluate agents' ability to understand human daily behaviors. Despite the recent success of large vision language models in many multi-modal tasks, complex situation reasoning over videos involving multiple human-object interaction events still remains challenging. In contrast, humans can easily tackle it by using a series of episode memories as anchors to quickly locate question-related key moments for reasoning. To mimic this effective reasoning strategy, we propose the Glance-Focus model. One simple way is to apply an action detection model to predict a set of actions as key memories. However, these actions within a closed set vocabulary are hard to generalize to various video domains. Instead of that, we train an Encoder-Decoder to generate a set of dynamic event memories at the glancing stage. Apart from using supervised bipartite matching to obtain the event memories, we further design an unsupervised memory generation method to get rid of dependence on event annotations. Next, at the focusing stage, these event memories act as a bridge to establish the correlation between the questions with high-level event concepts and low-level lengthy video content. Given the question, the model first focuses on the generated key event memory, then focuses on the most relevant moment for reasoning through our designed multi-level cross-attention mechanism. We conduct extensive experiments on four Multi-Event VideoQA benchmarks including STAR, EgoTaskQA, AGQA, and NExT-QA. Our proposed model achieves state-of-the-art results, surpassing current large models in various challenging reasoning tasks. The code and models are available at https://github.com/ByZ0e/Glance-Focus.
Paper
Similar papers
Peer review
Summary
This paper presents a model for video question answering called "Glance-Focus". The idea is to first "Glance" at a video to obtain a set of N dynamic memories, each represented as vectors of dimension D. The memories can either be learned in an unsupervised way or a supervised way. In the unsupervised case, the paper uses a collection of losses encouraging the video frames be assigned to different memory cells. In the supervised case, where there are ground-truth event labels that are localized in time, the paper uses bipartite matching to match the dynamic memories with the ground truth event labels, and that forms the basis of the loss. Next, the paper uses an encoder-decoder model. The encoder encodes the video features, memory cells, and question embeddings all together. Then, the decoder starts with a representation of the questio, and cross-attends to the memory cells, then to the video frames, and then to the answer. The model is initialized from S3D pretrained on HowTo100M, and learned on STAR, AGQA, and EgoTaskQA. The model seems to do well on all the datasets. -- update - I still agree with my concerns post-rebuttal and would like to keep my score.
Strengths
* The core idea that this paper seems to be tackling is an important one. Videos are long and redundant, and dealing with this space better is an important area of research. * The model seems to perform quite well on the STAR test set - (though, it should be noted, it was trained on the raw videos from that dataset, so it might be a bit of an unfair comparison versus the baselines).
Weaknesses
To this reviewer, it is rather hard to evaluate this work as the model and the associated losses feel very complicated. I am pretty sure I don't understand how they all play together. I think the paper could do a much better job: * Carefully ablating each one on a variety of datasets, extending Table 5 to consider all pairs of all losses. * Carefully ablating the modules and architecture choices. Table 4 seems like a good start, but it should be noted that removing the "focus" module for instance would presumably change the parameter count or effective footprint of the model. As a result it's hard for this reviewer to draw any conclusions there. It would also be great to explore a little bit the role of feature extraction. The paper uses S3D feature extraction which seems important. Are there any other methods on STAR that use S3D? Would it be possible to run other baselines? Next, I think this paper could be improved if it presented a method that could be learned across video datasets. The current method, at least to the best of this reviewer's understanding, has to be trained fresh on each one. As a result it might be overfitting to some of the dataset statistics of STAR/AGQA/EgoTaskQA and as a result absorbing some amount of human task-specific supervision from those sources. Finally it woud be great to not just consider STAR/AGQA/EgoTaskQA but also other video QA datasets like MSRVTT-QA, TVQA, etc. nit: L137-138: RoBERTa doesn't use WordPiece, it uses BPE tokenization
Questions
What do you see the role of each of the losses? Are all important, and could any be removed? Are there any architecture ablations that could be done while keeping the model size the same?
Rating
3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
nope, would be great to add such a section.
Summary
The paper proposes a novel approach for multi-event video question answering (VideoQA) that mimics the human reasoning strategy of glancing and focusing. The proposed method consists of two stages: glancing and focusing. In the glancing stage, the model generates a set of episode memories that summarize the key events in the video and supports both unsupervised and supervised methods, depending on the availability of event-level annotations. In the focusing stage, it uses the episode memories as memory prompts to guide the multi-level cross-attention between questions and videos, and predicts the answer based on the focused video frames. The paper evaluates the model on three challenging multi-event VideoQA benchmarks: STAR, AGQA, and EgoTaskQA. The experimental results show that it achieves state-of-the-art performance on all datasets, and outperforms current large vision-language models in various complex reasoning tasks. The paper also conducts ablation studies and visualizations to demonstrate the effectiveness of the proposed modules and memory prompts.
Strengths
1. The proposed method simulates the reasoning process exhibited by humans, namely an initial overview followed by a concentrated examination, thus presenting an interesting and novel approach. 2. The implementation is flexible and practical, enabling the simultaneous processing of annotated and unannotated videos without dependence on predetermined action vocabularies or event-level annotations. 3. This approach represents the pioneering effort to incorporate information maximization as a guiding principle for the training of unlabeled data within the realm of VideoQA. 4. It designs a multi-level cross-attention mechanism that leverages the episode memories as prompts to guide the model to focus on the relevant video frames for answer prediction. 5. The performance is quite impressive, as it even surpasses current large vision-language models in various scenarios.
Weaknesses
1. The writing has a large room to be improved. 2. In L166, the proposed method assumes there are C events in a video. However, it remains unclear whether C keeps consistent across all videos or dynamically varies for every single one. If C is predetermined, it would be valuable to conduct an ablative study to assess the impact of different choices for C on the overall performance of the method. 3. For unsupervised event generation, the authors impose a strong bias that the category labels should be evenly distributed across all memories. Further elaboration is essential to explain its reasonability. As from a subjective standpoint, only a limited subset of categories can manifest in a video given the contextual constraints (e.g., playing football on the ocean is impossible).
Questions
please refer to weaknesses.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
The limitations have been well discussed in supplementary materials. The potential negative societal impact is recommended to be addressed as well.
Summary
This paper proposes a two-stage method of constructing memories of periodical summarization of the video, then using memory prompts to local key frames for reasoning.
Strengths
1. The SOTA results on three Ego QA datasets. 2. The hierarchical construction of video abstraction might be a new way of tackling lengthy videos. 3. The IoU based un-unsupervised learning could be extended to other senarios.
Weaknesses
The writing needs improvement. Sentences are generally long and hard to understand. E.g., typo in abstract "abtain". Long sentence such as line 13-17. The idea of one-to-one matching of discrete events with memories seem to be outdated. Why not use soft-attention to assign soft weights of each event to the memory. This seems to be a well-studied technique. The dependency in a)-c) focus is not justified. Would this chain of dependency bring in heavy loss if M_enc is not accurate. The un-supervised construction of heuristic M seems to be not a suitable prompt. What would (9) be Q=M_foc + L_enc? Any ablation about this. The results seem not convincing in terms of the proposed focus chain. Would this accuracy improvement come from the additional parameters? Seems to be not a fair comparison with non-memory methods. Would methods like MIST. What would a baseline that does not construct M_enc explicitly but implicitly construct a M_foc do. Does the additional steps of constraining memory M from Eq2-5 harm the performance?
Questions
The results seem not convincing in terms of the proposed focus chain. Would this accuracy improvement come from the additional parameters? How is the positional embedding \phi formulated in Memory prompting? If it's from bi-partite assignment, is it trainable ? Not clear how IoU is in (4). Can you provide more details.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
NA
Summary
This paper aims to tackle the problem of multi-event VideoQA that involves multiple human-object interaction events. Inspired by the human's reasoning ability, this paper proposes the Glance-Focus model that can reason from coarse to fine. At the glancing stage, the Glance-Focus model can generate some dynamic key event memories to summarize the key events in the video. At the focusing stage, the Glance-Focus model can reason based on the most relevant moment through the cross-attention mechanism. The contributions of this paper are summarized as following: (1) This paper proposes a glance-and-focus mechanism to reason multi-event VideoQA in a human-like manner. (2) The Glance-Focus model achieves the state-of-the-art performance on some multi-event VideoQA dataset.
Strengths
(1) The idea of glancing and focusing on multi-event VideoQA is interesting and intuitive. This human-like design brings great potential for solving such multi-event VideoQA problem. The glancing can be seen as a kind of information compression to anchor key events, which may guide the focusing stage to conduct the fine-grained reasoning. (2) The motivation of unsupervised event generation at the glancing stage is clear. Because of the open-vocabulary world, it is hard to annotate all the events in the long-time video. To tackle this problem, this paper proposes to enforce the unsupervised constraints to generate meaningful key events. (3) To generate useful event memory in an unsupervised manner, this paper proposes to enforce the individual discrimination and the global diversity, which can guarantee to learn more meaningful key events. (4) This paper innovatively proposes to use event memories as anchors to prompt the model to locate the most relevant segments corresponding to the questions. Besides, the multi-level cross-attention mechanism can promote the event memory to focus on the question-related visual clues in a fine-grained manner. (5) Extensive experiments and analyses show the effectiveness and reasonability of the method. Specifically, the visualization of Figure 4 clearly shows the relationship between the event memory generation and the fine-grained focusing.
Weaknesses
(1) At the glancing stage, this paper utilizes the unsupervised learning solution to generate meaningful event memories. However, this paper lacks of the analyses about the inherent relations between the unsupervised learning (such as instance discriminative learning) and the key event memory generation. (2) The formulation of the temporal IoU loss (Eq.(4)) is not very clear and it should be explained more detailedly. (3) In experiments, some unsupervised results even outperform the supervised manner but necessary analyses about this phenomenon are lacking.
Questions
(1) The Glance-Focus model can not only tackle the problem of multi-event VideoQA but also tackle other kinds of reasoning problems. What about its scalability? The authors should provide more analyses. (2) In the unsupervised manner at the glancing stage, how to ensure the sequential order of events?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Yes, the authors have discussed the limitations in the supplementary material.
Summary
The paper introduces a model called Glance-Focus for Video Question Answering (VideoQA) tasks, which involve understanding human behaviors in videos. While vision language models have shown success in multi-modal tasks, reasoning over long videos with multiple human-object interaction events remains challenging. Humans can tackle this task effectively by using memory to quickly locate key moments for reasoning. To mimic this reasoning strategy, the Glance-Focus model is proposed. At the glancing stage, an Encoder-Decoder is trained to generate dynamic key event memories by aggregating relevant video frames. This is achieved through unsupervised memory generation using individual discriminating and global diversity loss, as well as supervised bipartite matching losses. At the focusing stage, the event memories enable the model to focus on the key event memory and then on the most relevant moment for specific reasoning using multi-level cross-attention. The model is evaluated on three representative VideoQA benchmarks (STAR, EgoTaskQA, and AGQA) and achieves state-of-the-art results, surpassing current large models in challenging tasks. The Glance-Focus model addresses the challenge of situation reasoning in VideoQA by generating dynamic event memories and using them to focus on key moments in videos.
Strengths
The paper's organization is good, it clearly explains the basic ideas. The paper proposed to involve memory to cross the gap between the concepts and video content in the QA transformers. The related work is organized well and will be better to introduce the recent benchmarks together with the VideoQA. The implemented experiments are sufficient and the example visualization is helpful.
Weaknesses
Firstly, in Table 1, AIO / Temp[ATP] / MIST results appear to be directly cited from [1] but seem to be based on the STAR validation part rather than the test part. Only RESERVE is reported on the STAR test. The dataset partition used for evaluation must be clarified for a meaningful comparison. In Section 4.1, please specify the average video length of EgoTaskQA. If available, additional statistics, such as the average number of events/action state changes per dataset, would be beneficial since the proposed method detects state changes in the 'glance' stage. This data will aid in understanding how the method handles scenes with varying event complexity. Furthermore, the paper lacks qualitative evaluation metrics for action/event/moment detection, such as IoU/mAP. These are essential for analyzing accumulated errors. The paper asserts the impracticality of using an action detector/event detector due to the close-set results it produces. However, a qualitative comparison with the proposed method using event/action detectors + rule-based filtering (keywords matching), e.g. Moment-DETR [2] and TubeDETR [3], is absent. Lastly, although the paper discusses multi-event detection, it still chooses multiple keyframes/visual clues in the focusing stages, which has been explored in previous works [4] and [5, 6]. A more comprehensive discussion comparing these related works would be appreciated. Some typos: Line 233 HowTo100m → HowTo100M. [1] D. Gao et al, MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answering [2] J. Lei et al, Detecting moments and highlights in videos via natural language queries [3] A, Yang et al, TubeDETR: Spatio-Temporal Video Grounding With Transformers [4] S. Yu et al, Self-Chained Image-Language Model for Video Localization and Question Answering [5] S. Kim, Semi-parametric video-grounded text generation [6] T. Qian et al, Locate before answering: Answer guided question localization for video question answering
Questions
Refer to the above sections.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
The codes should be open-source but it seems the paper doesn't mention
Summary
This paper addresses the task of question-answering in videos which have multiple events. The authors draw inspiration from the human capability to create event memories as a way to anchor the reasoning process over multiple events, and propose the Glance-Focus model, which aggregates a set of event memories (during a first "glancing" stage) and then proceeds to further process these memories ("focus" stage) to produce the final answer to the query. The authors benchmark their model on several standard multi-event benchmarks, such as STAR, AGQA (the balanced v2 version), and EgoTaskQA to characterize the effectiveness of their approach.
Strengths
`+` Question-answering in videos is an important task and challenging domain for multimodal models. Further, the goal of temporal reasoning is an important step for the development of AI agents in general. `+` The general direction of reasoning on top of generated/matched event memories draws inspiration from human event memory/reasoning, and seems like a reasonable approach. `+` The authors evaluation their technique on several benchmarks (representing different video distributions) and show improvements; I also appreciated that the authors looked at the v2 version of AGQA, for better characterization of the model capabilities.
Weaknesses
`-` The paper emphasizes "long video" understanding, but the length of the videos in the datasets examined are generally quite short (12-30 seconds) relative to many other video QA settings (e.g., ActivityNet-QA is ~2 mins). I understand this may be an inherited term from the prior work, MIST [10], which also uses similar terminology. However, that prior work did at least evaluate on NExT-QA, which is ~50% longer on average than the longest video dataset examined in this work here (AGQA). `-` Additionally, while the results on the short benchmark (STAR) are significant, the results become significantly more mixed when compared with closely related work (e.g., [10]) on longer benchmarks like AGQAv2 (+0.69 overall, and some specific categories like "sequencing", which seems like it should be better with event reasoning, are much worse than prior work). This also brings to question the efficacy of the proposed approach on longer video settings. `-` Further, the mixed results are even more striking due to the fact that this work leverages stronger low-level event supervision (for the bipartite event memory matching) than prior work (e.g., MIST) does. It is unclear how, without this level of supervision, the gains of the proposed technique, relative to prior work, are impacted (or the limitations of this approach as the videos become longer).
Questions
Overall, the rating for this paper pre-rebuttal is borderline. If the authors can address and clarify the points/questions raised in the weaknesses above, that would be helpful for updating the final rating and better grounding the discussion phase with the other reviewers.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors don't seem to provide a detailed discussion of limitations in the work, though the societal impacts does seem to be similar to related work in this space. It would be helpful and clarifying if there was an better indicator of how aspects of the proposed technique, e.g. the supervised bipartite matching, scale with longer videos, and how such limitations may be better addressed by future work. --- **Post-rebuttal update:** The authors largely address some of the key concerns that were brought up in the original review, in particular, the additional analysis on NExT-QA significantly improves the evaluation (the results on ActivityNet are promising, but not the *primary* focus of the updated review given the context and discussion below). While there are some still areas that can still be further improved (as detailed and described in the rebuttal discussion below), I am increasing my rating towards leaning acceptance since I believe that these are areas that can be further addressed by future work.
Kindly Reminder -- we are expecting the reviewer's reply to our rebuttal, thanks!
Dear Reviewer veHn, We appreciate your relatively positive review of our work such as motivation reasonable and suppose our paper has the potential to be given a higher score. Therefore we have tried our best to address every of your raised issues. We hope these will help clarify all your concerns and we are sincerely anticipating your feedback. We’d like to give a brief summary of our answers to your questions as follows. Motivated by your comments, we have conducted a comparative analysis of the current Video QA benchmarks in Tab.1 from the [Global Rebuttal](https://openreview.net/forum?id=J6Niv3yrMq¬eId=LQPJMIxMFG). Then we augmented the evaluation of our method on the NExT-QA and ActivityNet-QA datasets, which contain much longer videos. Even if these two datasets do not contain event annotations, our method can surpass the current SOTA methods (including MIST) without such supervision, as shown in Tab.2 and Tab.3 from the [Global Rebuttal](https://openreview.net/forum?id=J6Niv3yrMq¬eId=LQPJMIxMFG). Also, we have made further analysis of the event reasoning ability of our model on the AGQA v2 dataset as suggested in Tab.1 from the [Rebuttal](https://openreview.net/forum?id=J6Niv3yrMq¬eId=ePRr6BdSrO). The performance advantages of MIST mainly come from the binary answer questions which may show the overfitting phenomenon. However, our method demonstrates better results on the more difficult open-ended split. What's more, from the breakdown evaluation on the NExT-QA dataset in Tab.2 of the [Rebuttal](https://openreview.net/forum?id=J6Niv3yrMq¬eId=ePRr6BdSrO), our model also has strong event reasoning ability when scaling to longer videos. Warm regards, The Authors
thanks for the response! keeping my score
thanks for the helpful response and for running these experiments! I'm still pretty confused about this paper (as i was with my original review) - about what loss functions are effective here, what backbones/etc are most effective, and how much to trust the evaluation as it might be overfitting to dataset specific statistics. for this reason, I'd like to keep my score(3) - thanks!
We feel regret that our rebuttal failed to fully resolve all the concerns of Reviewer yVqG. Given the opportunity, we'd like to try again to summarize the experimental conclusions from our original [rebuttal](https://openreview.net/forum?id=J6Niv3yrMq¬eId=7Pgl2QPPl8) as follows. We sincerely hope such clarifications can offer further help. - **Loss Functions.** The loss functions we designed are shown in Tab.1 below. Only two loss functions are used in the supervised setting, $L_{L1}$ for temporal loss, $L_{cls}^{sup}$ for semantic loss. Due to the lack of event-level annotations in most cases, we **allow** unsupervised learning by **replacing** $L_{L1}$ with $L_{iou}$, **replacing** $L_{cls}^{sup}$ with $L_{cls}^{uns}$ & $L_{cert}$. **Table 1. Loss Functions in Glance-Focus** | | Temporal | Semantic | | ------------ | --------- | ---------------------------- | | Supervised | $L_{L1}$ | $L_{cls}^{sup}$ | | Unsupervised | $L_{iou}$ | $L_{cls}^{uns}$ & $L_{cert}$ | Referring to [Tab.1](https://openreview.net/forum?id=J6Niv3yrMq¬eId=7Pgl2QPPl8) in the original rebuttal, both temporal and semantic constraints are necessary. And in the supervised setting, the semantic constraint is more important, while in the unsupervised setting temporal constraint is more important. One can **flexibly** use these loss functions by adjusting their weighting coefficients. - **Backbones.** For fair comparisons, we have tried various backbones which are widely used in existing comparative methods. From [Tab.2](https://openreview.net/forum?id=J6Niv3yrMq¬eId=LQPJMIxMFG) and [Tab.3](https://openreview.net/forum?id=J6Niv3yrMq¬eId=7Pgl2QPPl8) in the original rebuttal, our method has consistently achieved SOTA results in both NExT-QA and STAR datasets with the same backbones as the compared methods. About what backbones are most effective, there is no consistent conclusion, and we deem it is somewhat less relevant to our proposed method. CLIP>S3D>C3D on NExT-QA. S3D>CLIP>ViT on STAR. - **Overfitting.** We thank the reviewer's insightful suggestion. Since our model can achieve SOTA performances across all **five** VideoQA benchmarks with various video lengths, different event numbers, diverse QA pairs, etc. We have quite strong confidence that our method has the potential for cross-dataset learning and will not overfit to a certain data distribution. We will further investigate this issue in our future study. - **Module Ablations.** Extensive experiments have been done in [Tab.1](https://openreview.net/forum?id=J6Niv3yrMq¬eId=hp0wWg88SQ) and [Tab.3](https://openreview.net/forum?id=J6Niv3yrMq¬eId=t7xsOOjmQ1) in the original rebuttal. We believe the effectiveness of the proposed glance module and focus strategy has been fully verified with **consistent** parameters.
Discussion
Dear Reviewers, Thank you for reviewing this paper. Authors have provided their rebuttal. Would you please check it, and give your comments/rating based on the rebuttal letter and the comments from other reviewers? Best Regards AC
Clear response and a solid work, keep the original rating for accpeting.
Thanks for the authors' clear response! The motivation of unsupervised event memory generation is clear. It is enlightening to use information maximization to generate meaningful memories in this task. The explanation about the case of "Unsupervised>Supervised" sounds reasonable and I suggest the authors that they should add these detailed analyses in their next version. Lastly, it is meaningful that GF model shows comparable performance with other action detection models. I suggest the authors that they should further extend this idea to other video understanding tasks. Based on the above analyses and other reviewers' postive comments, I keep the original rating to accept this maniscript.
We‘d like to thank you again for your affirmation of our work. We are very glad that our response can address all your concerns. We will incorporate the detailed analyses and extended results in the revised version and we are trying to extend our work to the video captioning task. If you have any further questions or concerns, welcome to discuss with us. Your feedback is greatly appreciated!
Thank you for raising the score! Your suggestion is very enlightening to us. We are very glad that our further analysis can address all your concerns. We will accordingly revise our next version to clarify such issues.
Thank you for raising the score! We are glad that our response has resolved all your concerns. The relevant analyses and experimental results will be updated in the revised version.
Feedback
The responses answered most of the questions and provided sufficient experiments. Thank you for your efforts.
Thank you for the rebuttal
Thank you for the rebuttal! I think the updated results/detailed analysis is promising, and feel the rebuttal has addressed many of my key concerns. A couple of additional notes: **ActivityNet-QA results (clarifying claims and details)** As a clarification, my understanding is that this model is actually *not* state-of-the-art on ActivityNet-QA, counter to what is suggested in this part of the global author rebuttal: > Furthermore, as Reviewer VeHn & yVqG suggested, to verify our model on more VideoQA datasets, we additionally test on longer video understanding benchmarks, ActivityNet-QA. The performance shows that GF also achieves SOTA result and even surpasses the Just-Ask model pre-trained on large VideoQA data. The main comparison with JustAsk (ICCV 2021) is helpful, but there are more relatively recent works with much higher accuracies. As one example, FrozenBILM from last year's NeurIPS 2022 (see `[M1]` below), which achieves `43.2` (see Table 6, "w/o speech"). I think the provided results are still promising, but *the claims should be made more accurate/calibrated* (right now, the statement seems like it is unequivocally the SOTA across all prior work, rather than perhaps an improvement on some specific prior model/feature setting in prior work). Related to the above, I think I have a general idea of what settings were used based on prior work, but if the authors can provide a *confirmation* of details on what were the specific features/model/task protocol/etc details, that would be helpful too. For example, specifying the relation to the JustAsk baseline, which used frozen S3D-HowTo100M features (which I assume is likely the same as ones used in this work, per L232-233 in the original submission, but possibly some other features like CLIP also?) + some specific protocols. `[M1]` Yang et al. "Zero-Shot Video Question Answering via Frozen Bidirectional Language Models." NeurIPS 2022. **AGQAv2/NExT-QA results (additional general comments)** The AGQAv2 analysis is helpful, and better illustrates the difference between this method and MIST. This analysis, combined with the NExT-QA experiments, really help to flesh out the core empirical support for the work. The NExT-QA results are promising, and it is good to see both the overall results and the detailed breakdown on C/T/D compared with MIST (especially, showing a modest improvement on C, and significant one on T). As an additional suggestion from recent/concurrent work (for a later revision, not necessary for now), it would be interesting to see if the performance improvement deltas increase further when compared on the "hard" C/T subset from [3]. **Overall:** I appreciate the effort the authors put for the rebuttal, and feel it has made an overall net positive impression on the work (of course, assuming the claims/details/etc will be clarified in a later revision). I will plan to update my final review and rating during the final reviewer-AC discussion phase.
We are glad that our response can address your concerns. We will update the additional experimental results, detailed experiment settings in the later revision, and modify the claims in the experimental analyses. Explanations on your additional notes are as follows. **ActivityNet-QA results** Thanks for the reviewer's kind reminder. We will revise our statement to make our claim more precise. Considering that ActivityNet-QA does not belong to the typical multi-event video QA benchmark, we only showed a concise experimental analysis for the limited space. We further explain the details of the experimental setup under this dataset. From the observation in Tab.2 of our [Global Rebuttal](https://openreview.net/forum?id=J6Niv3yrMq¬eId=LQPJMIxMFG) on the NExT-QA benchmark, GF model with CLIP feature is better than C3D/S3D feature. So we exploit CLIP feature on the more challenging ActivityNet-QA with longer videos, by using only QA labels as supervised information from the train split of the dataset and evaluating on its test split. The Just Ask baseline used the frozen S3D-HowTo100M video features. The specific comparison is shown in Tab.1. As for your mentioned recent work FrozenBiLM, it is yet another model pre-trained on the large dataset WebVid10M, which is consist of 10 million video-text pairs and thus significantly benefits the final accuracy. **Table 1. Detailed comparision on ActivityNet-QA.** | Method | Pre-training Data | Training Data | Features | Acc on ActivityNet-QA | | ------------------------------------------------------------ | ----------------- | -------------- | -------- | --------------------- | | [HGA](https://ojs.aaai.org/index.php/AAAI/article/view/6767) | - | ActivityNet-QA | C3D | 34.6 | | [LocAns](https://arxiv.org/abs/2210.02081) | - | ActivityNet-QA | C3D | 36.1 | | [Just Ask](https://arxiv.org/abs/2012.00451) (w/o pre-trained) | - | ActivityNet-QA | S3D | 36.8 | | GF-CLIP(uns) | - | ActivityNet-QA | CLIP | **41.1** | | [Just Ask] (pre-trained) | HowToVQA69M | ActivityNet-QA | S3D | 38.9 | | [FrozenBiLM](https://arxiv.org/abs/2206.08155) | WebVid10M | ActivityNet-QA | CLIP | 43.2 | Considering the fairness of the comparison, we did not compare the performance with more large pre-trained VL models. Similar to FrozenBiLM, these models all use additional large-scale data to train the networks. By designing different pre-training tasks, this type of models can achieve very good results on single-event video QA benchmarks, such as ActivityNet-QA. For more results, please refer to the [leaderboard](https://paperswithcode.com/sota/video-question-answering-on-activitynet-qa) of ActivityNet-QA. On the contrary, we focus on proposing a strong multi-event video QA model that does not rely on large-scale pre-training data. Nevertheless, as discussed in "Generalization of the Model" from the [rebuttal](https://openreview.net/forum?id=J6Niv3yrMq¬eId=7Pgl2QPPl8), our method can also be extended to a pre-training model based on video-text pairs, which will be considered in our future work. **NExT-QA results** We are glad that our analysis on NExT-QA dataset is helpful. As the reviewer suggested, we tried our best to further evaluate our model on the ATP-hard subset of this dataset in this last day of the Author-Reviewer discussion phase. Due to the extremely limited time, we are only allowed to make a brief comparison with the methods that have reported the results on this subset in their original papers. A more complete comparison with methods such as MIST will be updated in our revision. From Tab.2, our model is shown to still achieve strong performance on both hard Causal and Temporal questions compared to models with the same video features. Among the comparative methods, CoVGT demonstrates superior accuracy, by additionally involving the fine-grained object-level modeling with Faster R-CNN video features. **Table 2. Evaluation results on ATP-hard subset of NExT-QA.** | Methods | Features | Acc@C | Acc@T | | ----------------------------------------- | ----------------- | -------- | -------- | | ATP | CLIP | 19.6 | 22.6 | | Temp[ATP] | CLIP | 38.4 | 36.5 | | HGA | C3D | 43.3 | 45.3 | | [CoVGT](https://arxiv.org/abs/2302.13668) | CLIP+Faster R-CNN | **51.8** | 50.5 | | GF-CLIP(uns) | CLIP | 48.7 | **50.8** |
Thank you for the additional response
Thank you for the detailed follow-up message! Once again, I appreciate the efforts of the authors. - On ActivityNet, the additional details are helpful for contextualizing the results (e.g., use of CLIP vs. S3D, etc.), and the points raised by the authors are reasonable (e.g., ActivityNet is a "concise experimental analysis" outside the standard multi-events benchmarks set). It would still be interesting (as part of _optional_ future work) to see how GF performs with the same frozen S3D features, since these were used in other parts of this work (and are comparable to what is used in "JustAsk, w/o pretrained"). Overall, I do think the results are promising, even if they are not the absolute state-of-the-art, and I am glad that the authors will plan to include the additional necessary context and edits to their claims in a revised paper/supplement. - On NExT-QA, the updated evaluation is helpful and the results seem to be good, especially considering extra object features were not used for GF. I understand the additional comparison with MIST will take some time for the reasons mentioned, and in particular, will require more time than the scope of the rebuttal period -- this was expected and why I mentioned "(for a later revision, not necessary for now)" in my earlier response. Nonetheless, seeing the preliminary result now in a short time frame is a pleasant surprise and is also helpful. I'm still planning to finalize and update my review/rating during the final discussion, but in case it was not clear earlier, I do plan to increase my rating based on the author rebuttal.
Decision
Accept (poster)