SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM

Summary

This paper focuses on the video LLMs (Vid-LLMs) area and attempts to improve the fine-grained video understanding. The authors design modules and training data to identify relevant temporal segments based on the given query and then generate outputs based on the identified segments.

Strengths

Overall, the writing is well-organized and easy to follow. The strengths can be summarized as follows: 1. This work introduces a two-stage inference strategy into the Vid-LLMs area, following the coarse-to-fine mode. Such a strategy provides a new perspective for enhancing video understanding. 2. This work proposes a new benchmark FineAction-CGR to assess the ability of Vid-LLMs to process fine-grained temporal understanding tasks, which provides a new perspective to evaluate Vid-LLMs' capabilities.

Weaknesses

There are some weaknesses listed as follows: 1. Although the proposed coarse-to-fine reasoning approach is relatively novel in Vid-LLMs, I have concerns about its practicality. Firstly, this reasoning method nearly doubles the inference cost for the same question compared to other methods. Secondly, the accuracy of the final response heavily relies on the accuracy of event localization in the first stage. According to the benchmark validation provided by the authors, even when trained on proprietary data, SlowFocus only barely meets the passing mark in temporal grounding mIoU. This significantly impacts the accuracy of temporal reasoning in the second stage. 2. The authors chose relatively weak baselines and attempted to claim the superiority of their results. I suggest comparing their method with MovieChat [1] and VideoChat2 [2] to substantiate their claims of advancement. Moreover, despite using more data than LLaMA-VID, their performance on various benchmarks is not significantly better than LLaMA-VID, raising questions about the effectiveness of this two-stage approach. 3. The authors need to validate the effectiveness of their two-stage model on benchmarks involving long videos, which is crucial to substantiate their claims. For instance, they could use benchmarks like EgoSchema and MovieChat-1K for this purpose. [1] MovieChat: From Dense Token to Sparse Memory for Long Video Understanding. CVPR 2024 [2] MVBench: A Comprehensive Multi-modal Video Understanding Benchmark. CVPR 2024

Questions

See weaknesses above.

Rating

4

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

This paper focuses on fine-grained video understanding with large language models. Current works suffer from the dilemma between the per-frame token number and temporal sampling frequency to maintain an acceptable sequence length into the language model. The authors propose to sample a global view with low sampling frequency to perceive the temporal positions that are related to the question, then sample the frames within the predicted temporal window with high frequency for fine-grained video perception. The proposed dataset FineAction-CGR can well reflect the ability in temporal grounding and fine-grained question answering.

Strengths

1. The exploration in the per-frame token number and temporal sampling frequency for video understanding is important and fundamental. 2. The authors propose to first perceive the related temporal window and then densely sample the frames within the temporal span. The architecture makes sense and is suitable for fine-grained video captioning and question answering while maintaining computation efficiency. 3. The dataset can evaluate the temporal grounding and fine-grained video understanding ability and could make contribution to the progress in video understanding with large language models.

Weaknesses

1. How is the duration of the videos for temporal grounding evaluation? Will the low frequency sampling result in the $h_L$ to lose much information in excessively long videos? 2. In Fig.2, the temporal grounding derives from the low frequency sampled video features. Why the high frequency sampling also has impact on the temporal grounding performance? 3. The architecture of the temporal encoder is not illustrated. 4. In Table 5, what does Fps=64 mean? The video dataset usually contains videos with around 30 fps, how does 64 fps work? 5. The proposed method presents overwhelming advantage on the proposed FineAction-CGR dataset while the improvement on other benchmarks are marginal. I understand it is because the rest are comparatively easy and of coarse-grained. However, it is better to provide the evaluation on other challenging video benchmarks like long video datasets, e.g., EgoSchema [1], MovieChat [2] to validate the effectiveness of the temporal grounding in long sequences for more comprehensive comparison. [1] Mangalam, Karttikeya, Raiymbek Akshulakov, and Jitendra Malik. "Egoschema: A diagnostic benchmark for very long-form video language understanding." Advances in Neural Information Processing Systems 36 (2024). [2] Song, Enxin, et al. "Moviechat: From dense token to sparse memory for long video understanding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Questions

How do you condense each frame into $N_v$ tokens?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed the limitations on the resolution.

Summary

This works proposes “SlowFocus” to improve the balance between *tokens per frame* and *frames per video* used in Video LLMs. SlowFocus identifies video segments relevant to a given query and samples selected segments at high frequencies. The high frequency tokens are mixed with low frequency global video tokens. Authors propose suitable training strategies to learn the newly introduced layers. A new evaluation benchmark is also proposed by the authors.

Strengths

1. Motivation for work is well established 2. Clear explanation of proposed methodology 3. Evaluation benchmark contribution NOTE: Is the proposed dataset released publicly?

Weaknesses

1. **Motivation not justified:** The idea is to improve video LLM performance at a fixed compute. However, there are no results to evaluate the inference speed of proposed SlowFocus with prior works. Hence, the benefit of this approach over uniform sampling at a higher resolution is unclear. 2. **Table 1 Unfair:** It appears that prior works are zero-shot while SlowFocus is trained on this FineActionCGR dataset. This makes the comparison unfair. 3. **Table 2 Missing Details:** Is SlowFocus re-trained on this data? How many frames do the prior works use? What is their inference compute requirement in comparison? Also, can a simple VLM + LLM baseline like LLoVi [1] be added for comparison? This will give a better idea on usefulness of proposed method. 4. **Missing Ablations:** How does this compare against simply feeding the selected high-res frames and the global low-res frames (arranged in temporal order) into an existing Vid-LLM (e.g. LLoVi)? 5. [Minor] Related work [2] using spatial coordinates for same video tasks [1] LLoVi: https://arxiv.org/pdf/2312.17235 [2] Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs, CVPR 2024

Questions

N/A

Rating

6

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

Addressed

Summary

This paper designs a SlowFocus mechanism to allow Vid-LLM's input signals to combine both high frame rate and low frame rate inputs simultaneously. This addresses the issue of maintaining the effectiveness of input information within a limited context window in LLMs. Low frame rate inputs contain global information, while high frame rate inputs contain local details. Additionally, this paper proposes a Multiple-Frequency Mixing Attention mechanism to better integrate these inputs and a Temporal Relationship Modeling mechanism to preserve temporal relationships in the temporal dimension.

Strengths

1. The SlowFocus mechanism, which combines high frame rate and low frame rate inputs, is highly reasonable and efficient. 2. The Multiple-Frequency Mixing Attention mechanism and Temporal Relationship Modeling are appropriate technical designs. 3. The proposed three-stage training and corresponding dataset construction are rational. - The use of temporal grounding data for the second stage of training followed by video level instruction fine-tuning data for the third stage is highly instructive. 4. The FineAction-CGR benchmark fills the gap in the community for fine-grained video understanding evaluation. 5. The experiments in this paper are comprehensive, effectively demonstrating the validity of each design point. - The writing is clear and easy to understand.

Weaknesses

1. Line 126: Using the same letter "L" to represent both "Low" and "LLM" might appear confusing to readers. 2. Table 5: There is some confusion regarding whether this pertains to the low frame rate parts or the high frame rate parts. 3. Line 449: - What is the data ratio processed by GPT-4V and the Video Recaptioner Model? - Do their results show significant distribution differences? - Are you using GPT-4V to generate frame-by-frame videos or video captions? 4. The analysis on the effectiveness and rationality of the FineAction-CGR benchmark evaluation is insufficient.

Questions

1. Why is the visual encoder turned off in all stages? Some recent work also start open the gradient of CLIP encoder, would the results improve in your model if it were turned on? 2. Is the use of LoRA for pre-training due to resource constraints? Would full-rank training result in better performance?

Rating

7

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

It is recommended to include comparisons with closed-source models such as Gemini Pro, GPT-4V, etc., to provide reference scores for FineAction-CGR benchmark in Table 1.

Answer to the authors

This work is designed to be more geared towards long video understanding. Otherwise for short videos, so-called event localization and video slicing wouldn't make much sense. However the authors didn't provide convincing results on long video benchmarks, and I needed comparative results on EgoSchema and Video-MME to see exactly how SlowFocus performs.

Thanks for response.

The author has addressed most of my concerns. The ideas presented in this paper are quite ingenious. I do not believe that the combination of high and low-resolution inputs poses a computational efficiency issue. The author's strategy effectively addresses the challenge of long video understanding. I stand by my rating.

Concerns unaddressed. Maintaining 'reject' rating.

Several issues were raised my reviewers and post-rebuttal most of these remain insufficiently addressed by the authors. 1. **Motivation not justified:** The first contribution, to quote the paper, is *"to resolve the prevalent trade-off in existing Vid-LLMs..."*. This trade-off is only valid at a fixed inference compute budget - the original paper had no discussion / evaluations regarding inference time compute. The authors provide two sets of new tables including inference time and tokens per frame in the rebuttal, however omitting important details on what dataset these results are on, compute used (GPUs) to benchmark inference times, how the baselines were implemented, and why the baselines are not at all competitive (baseline numbers significantly lower). More important, the concern of "benefit of this approach over uniform sampling at a higher resolution" remains unaddressed. Uniform sampling at a higher resolution (i.e. larger FPS) should be able to achieve the same effect of authors method and possibly perform at similar compute for the kind of short videos used in evaluation tasks. This is not discussed at all, and the motivation for using the authors proposal over this default (current norm papers cited in original review) is unclear. 2. **Unfair results comparisons:** The new results in rebuttal (LLaMA-VID mIoU increases from 0.35 -> 22.38 with fine-tuning) further support how baselines used as comparison in the paper were unfair. The authors introduced a new task and dataset (that baselines were not built for) and compared a fine-tuned version of their method with the zero-shot baselines. While the authors newly introduce results where baselines are fine-tuned in the rebuttal, details of baseline fine-tuning is unclear (their results maybe suboptimal due to unsuitable fine-tuning of those methods). Also, only a single baseline is provided under these slightly fairer fine-tuning settings for comparison. The unfair results comparison issue still remains unaddressed. 3. **Video QnA results:** The results on activity-net provide a better comparison. However, the baselines used are not the best currently. See https://paperswithcode.com/sota/zeroshot-video-question-answer-on-activitynet or https://arxiv.org/pdf/2403.04640 (ECCV '24). Also, agreeing with the sentiments of reviewer bGgU on need for some long-video QnA benchmarks. 4. **Missing ablations:** The authors avoid providing the requested ablations. In original review, frequency = temporal frequency (i.e. frame rate). The original request was not for "one that feeds selected high-frequency frames" into a baseline, but feeds all frames at that high-frequency (and concern 1 is how authors method is better than such a baseline). NOTE: Is the proposed dataset publicly released? This is unclear. The idea and method of the authors is interesting and could be valuable. However, in the current form, the paper lacks sufficient evaluation to verify its effectiveness and usefulness. Hence, voting to reject this paper.