Summary
This paper proposes VideoStreaming to understand arbitrary-length videos by streamingly encoding video tokens and adaptively selecting a constant number of them. Through careful designs, including a small LLM streaming encoder, prefix task, modified attention, memory propagation, and Gumbel-Topk token selection, the method successfully achieves superior performance on long video benchmarks, showcasing precise temporal comprehension for detailed question answering.
Strengths
1. This paper has a strong motivation, aiming to solve the significant computational burden caused by the large number of tokens extracted from long videos. This is an important issue for LMM to understand long videos.
2. The methodology is novel. First, the paper utilizes the causal LLM to satisfy the streaming and query-related vision summarization with explicit timestamp cues. Second, the paper designs a prefix task that naturally compresses the vision tokens during auto-regressive training. Lastly, through memory propagation and adaptive selection, the model successfully uses only the contextually related tokens to forward to the LLM, thus reducing the computational and memory cost due to the long kv cache.
3. The paper is well written and easy to follow.
Weaknesses
My main concerns are as follows:
A. In common short/medium video length benchmarks, VideoStreaming could be unnecessary;
B. For long video length benchmarks, the paper does not provide enough experiments;
C. Promising experiments, like real-time interaction, are completely ignored in this paper.
Specifically,
A.1. This paper actually uses much more data during training compared to LLaVA-Next-Video:
- LLaVA-Next-Video-7B: 558K for MLP + 860K for MLP & LLM
- VideoStreaming-8.3B: 790K for MLP + 763K for small LLM + 345K for all
VideoStreaming is trained with 480K more data, which cannot be ignored. However, note that LLaVA-Next-Video-7B has slightly better results on VideoChat-GPT and even employs a much simpler pooling strategy to encode video tokens. Given that VideoChat-GPT's average duration is 1.81 minutes (see Table 1), we can infer that when the short video is less than 2 minutes, VideoStreaming may not offer advantages.
A.2. For medium-length videos, such as EgoSchema (average 3 minutes), the paper seems to have missed the results of VideoChat2 mistral-7B, which shows an accuracy of 54.4%, much higher than the 44.1% reported in the paper. Although it is not entirely fair since VideoChat2 mistral-7B uses even more training data, its model employs a simple Q-former with 16 segments and 16 frames to handle the video. Therefore, I also suspect that VideoStreaming may not present advantages in medium-length videos. Here I have omitted the discussion of MovieChat-1K since its question setting does not really relate well to minute-long videos.
B. Therefore, where VideoStreaming might truly excel is in handling very long videos. However, it is very disappointing that the paper does not present enough experiments on this. The ablations have been done on medium-length videos, and the authors do not provide any competitive baselines (with the same training data and procedure) in Table 7.
C. In fact, VideoStreaming completely ignores a promising application, which is long-term, real-time streaming QA, such as GPT-4o (concept). The method can significantly reduce the kv cache consumption during long video streaming and maintain high-speed response even if a long sequence has been input.
D. The demo shown in Figure 5(b) can be misleading. If you try the text-only Vicuna 13b (https://chat.lmsys.org/) and ask, "What challenges does Philippe Petit face in his quest to walk between the Twin Towers?" it can produce even better responses, such as "getting access to the top of the Twin Towers," "creating the perfect wire," "practicing the walk," and "keeping the walk a secret." This suggests that the question you provided may only prompt the LLM to recall its text pretraining data. I also doubt that the model can precisely select the relevant clips in a 2-hour movie, as that would be an extraordinary feat even for GPT-4o or Gemini-1.5-Pro.
Despite the above weaknesses, I still acknowledge the novelty of the methodology. I will decide on my final rating after reading the rebuttal.
Questions
1. Would it be possible to use the previous layers of Vicuna to replace the extra introduced (half) Phi-2, which has 1.3B parameters that cannot be ignored?
2. Should L127-L129 be correct? StreamingLLM [a] has demonstrated the importance of initial tokens. I think it is your designed prefix task that restricts the streaming encoder from achieving this.
3. The method still needs to segment clips. Why not directly operate on the frames with a moderate FPS (e.g., 2)? This should make the method more general. Or will this also incur a huge memory cost for the streaming encoder since it is still at the billion-level?
4. The ablation in Table 8 is hard for me to understand. First, what is the performance if there is no memory and no selection? Second, the performance variance in MovieChat is so huge. Are these results from different trained models, or just the same trained model but different inference?
[a] Efficient Streaming Language Models with Attention Sinks, ICLR 2024.
Limitations
The limitations are discussed in the supplementary material: (1) simply uniformly sampling frames to form a set of short clips for memory-propagated streaming encoding rather than using adaptive segmentation; (2) the number of tokens may not be sufficient to represent clips with abundant visual content and intensive temporal dynamics. These are indeed limitations and are well discussed. However, as mentioned in the 'weakness' section, most limitations pertain to the experimental setting.