Summary
This paper introduces VideoTree, a framework that offers a dynamic hierarchical video representation enabling LLMs to reason over long videos.
The method has 3 stages:
- The first stage—adaptive breath expansion—utilizes K-means to cluster visual features extracted from video frames, a captioner to obtain text descriptions, and an LLM to assign a relevance score for each cluster.
- These clusters are expanded according to their relevance scores in the next stage, with the most relevant clusters expanding into two-level trees.
- Finally, in the last stage, this tree is traversed top-to-bottom in temporal order to obtain a textual description of the video, which is then fed into the LLM alongside the query. Intuitively, tree representation aims to reduce the high redundancy of information in video data while preserving the fine-grained details that are relevant to the query.
The authors claim that VideoTree outperforms non-hierarchical methods such as VideoAgent and LLoVi on EgoSchema and NExT-QA benchmarks, obtaining a result comparable to LVNet, which capitalizes on the stronger GPT-4o backbone.
On Video-MME benchmark, which features long videos up to 1 hour in length, VideoTree slightly outperforms the proprietary GPT-4V model but comes up short against GPT-4o and Gemini 1.5 Pro.
Against 6 open-source MLLMs that were extensively trained on video data, VideoTree defeats 4 of them (Table 2) despite being a training-free approach.
Strengths
Although hierarchical video representation existed before this paper, they operated in a bottom-up manner. In contrast, the proposed method improves efficiency and effectiveness by employing a top-down approach with dynamic depth. Thus, the method exhibits sufficient novelty. By surpassing the performance of previous training-free methods, the method attempts to demonstrate its significance (However, it falls short against state-of-the-art MLLMs, both proprietary and open-source.)
The paper explains the method clearly, with diagrams visualizing the information flow. The main text provides a concise overview, which is supplemented by the appendix that provides further details.
Weaknesses
Throughout the paper, the authors claim that both redundancy and irrelevance of the information is harmful (e.g., in line 10, "Long-form video understanding has been a challenging task due to the high redundancy in video data and the abundance of query-irrelevant information."). Although it looks easy to intuitively recognize the detrimental effect of irrelevant information, the claim about redundancy is not well-substantiated, especially in terms of empirical proof. In fact, one could argue that some of the results constitute empirical proof against this claim. Specifically, Figure 3 demonstrates that increasing the number of captions improves performance for both LLoVi and VideoTree.
Due to temporal redundancy in videos, more captions translate to more redundancy, which should be detrimental according to the authors, yet this seems to improve the accuracy. Moreover, the proposed method—despite filtering out irrelevant content—introduces more redundancy in a different form because of the tree structure. Given that reducing redundancy is a major motivation behind the proposed method, authors should justify their claim about redundancy, either by citing relevant work (if available) or providing empirical proof.
In line 30, the authors write "VIDEOTREE framework achieves better performance than the strong proprietary GPT-4V model and other MLLMs that were extensively trained on video data." I think this sentence seems to suggest VideoTree outperforms all tested MLLMs, which does not seem to be the case, as shown in Table 2. The authors should clarify this point by replacing "other MLLMs" with "many other MLLMs", for example. Overall, the authors should make it clear that the proposed method does not outperform the best proprietary and the best open-source MLLMs.
Although the fact that it can outperform numerous MLLMs despite being a training-free method is a technical feat, the paper doesn't explain the advantages of this method over an off-the-shelf MLLM. In other words, given that the training cost of MLLM has already been paid, why should we use a training-free approach? Therefore, more justification as to why this training-free method is preferable to using a pretrained MLLM would help emphasize the importance of the method. For instance, the authors could show that MLLMs require more computation and longer inference times.
Finally, the paper does not consider the possibilities for future work. The authors can address this by briefly mentioning some ideas in the conclusion section. This consideration can also improve the paper's position within the literature, thereby highlighting its significance.
Minor writing mistakes:
- Space before dot at line 128.
- Line 285: "recent-proposed" should be "recently proposed".
Questions
1. In line 450, the authors write "...VideoAgent baseline, which suffers from performance degradation after 11 frames..." However, the x-axis in Figure 3 represents the number of captions. Did the authors mean "captions" instead of "frames"? Are they referring to the number of captioned frames?
2. Line 450: "our method continues improving, generalizing to 62.4 frames". What is the meaning of this fractional frame count? Assuming that they are referring to the number of captioned frames, the question still stands. Was this value averaged over video samples? Did this situation arise because some videos are shorter?
3. The caption of Table 8 states that VideoAgent's avg. LLM calls are estimated. Why weren't real values used? How were they estimated?
4. The prompts in Table 14 and 15 include queries about confidence. How are these confidence values used?
5. What is the reasoning behind the FPS choice (1 FPS for EgoSchema and NExT-QA, 0.125 for Video-MME)? How would the performance change if the FPS were changed?