Artemis: Towards Referential Understanding in Complex Videos

Summary

This paper introduces Artemis, a video-language model for video-based referential understanding. It can describe a target referred by a bounding box in a video. A referential video dataset named VideoRef45K is collected to train the model. Artemis is evaluated on HC-STVG benchmark and outperforms baselines adapted from image-based referring models. Besides referential understanding, Artemis can also perform general video question answering, and serve as a component in multi-round and long-form video understanding.

Strengths

1. A video-based referential understanding dataset, VideoRef45K, is established. It facilitates the development of the area, providing referential pretraining data. 2. The design of target-specific feature branch in the model architecture is well-motivated. 3. Although no video-based referring model exists, the paper adapts image-based referring models to video as baselines. The adaption method is quite reasonable. 4. Artemis outperforms the adapted image-based baselines significantly on HC-STVG. 5. Artemis can still perform general video question answering and achieve better performance after training on the video referring task. This demonstrates that video referring can boost the reasoning capability of video-language models. 6. Combined with existing video-language models, Artemis can perform multi-round video understanding with grounding and long-form video understanding.

Weaknesses

1. The RoI selection step of Artemis clusters the object bounding boxes from different frames. However, the clustering algorithm only considers the bounding box coordinates but does not take the visual content in the bounding boxes into account. In some cases, the bounding box of an object may remain unchanged for a long time but the object state keeps changing, e.g., a person standing at a certain location performs a series of actions. Clustering these frames together can compressing them into one would lose valuable information. 2. The proposed Artemis is built based on video-language models Video-LLaVA and Video-ChatGPT. However, these video-language models are not used as baselines in the experiments. Although they are not originally developed for video referring, there is a simple approach to adapt them for the task. As suggested by [1], directly drawing a circle or a rectangle on an image can help VLMs focus on the indicated object. Therefore, one can adapt the video-language models for video referring by drawing the object bounding boxes on the video frames and ask the model "What is the target indicated by the red rectangle doing?". Artemis should be compared with this simple baseline to demonstrate the effectiveness of the RoI feature branch in its model architecture. [1] Shtedritski et al. What does CLIP know about a red circle? Visual prompt engineering for VLMs. ICCV 2023.

Questions

1. Does the target-specific features in \<region\> tokens include the positional information of the bounding boxes?

Rating

7

Confidence

5

Soundness

2

Presentation

4

Contribution

4

Limitations

Limitations are discussed in the paper.

Summary

This paper introduces Artemis as a robust solution for the video-based referential understanding task. This task involves analyzing complex videos, each spanning 20–30 seconds, where the target performs multiple actions. Given a video, the Multimodal Large Language Model (MLLM) attempts to answer questions such as "What is the target <region> doing in this video?" with <region> referring to a bounding box in any video frame. Artemis follows the general design principles of modern MLLMs, such as visual instruction tuning. To extract target-specific video features, Artemis employs a straightforward yet effective approach involving (i) tracking the target over time and (ii) selecting informative features from a comprehensive list of regions of interest (RoIs). The training of Artemis consists of three stages, with the first two stages being similar to LLaVA. For the final stage, this paper introduces the VideoRef45K benchmark, comprising 45,000 video question-answer pairs, with box-level prompts and answers for complex videos. Experiments demonstrate the promising performance of Artemis across various quantitative metrics, including BERT score, BLEU, and more.

Strengths

This paper is well-written and easy to follow. The authors present a straightforward solution to address video-based referential understanding, a relatively unexplored research area so far as shown by the authors. The method adopted is simple yet aligns well with the paper's motivation. Additionally, the experimental section is robust and well-executed.

Weaknesses

1. As far as I know, Multi-Object Tracking (MOT) is far from satisfying in accurately tracking target regions, particularly in challenging scenarios such as motion blur and occlusion. Although this paper mentions these issues in its limitations section, it does not discuss them in detail. My concern is that the method performs well on the presented benchmarks because the scenarios to be tracked are relatively simple. It would be beneficial if the authors could provide the performance of HQTrack on these benchmarks or offer examples showing the method's efficacy in complex scenarios (e.g., multi-person tracking). 2. This paper does not delve deeply into learning the temporal relationships among tracked Regions of Interest (RoIs). Therefore, the temporal encoding knowledge of these RoIs mainly derives from HQTrack. Since the improvement of this model in video-based referential understanding primarily stems from its better comprehension of temporal dynamics, it raises the concern that the benefit may come from HQTrack rather than the model itself. 3. Comparisons with other models (e.g., Shikra, Merlin) seem somewhat unfair, as these models do not utilize the dynamic knowledge from HQTrack. A fairer comparison setting would enhance the validity of the results. I am willing to upgrade my rating, if the authors can address my above concerns.

Questions

I am a bit surprised that MLLMs for video referring and grounding have not been explored yet. As I am not very familiar with referential understanding, I look forward to more feedback from other reviewers. If this paper does not overlook any important references, I am willing to upgrade my rating.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Please refer to weakness.

Summary

The paper proposes to bring fine-grained understanding to multimodal LLMs (MLLMs) by introducing video-based referential understanding task. The paper motivates with the drawbacks of current image- and video-based MLLMs, and the need for region-specific features to answer region-specific questions . The proposed approach expands the set of video-level features with target region-specific features (box-level prompts) via tracking (HoITrack), alignment (RoIAlign) and selection (clustering). The paper ablates the effectiveness of tracking and selection criteria for improved performance.

Strengths

- The paper is well-written, clear, and the motivation for target-specific features is well presented

Weaknesses

- A fair baseline - It’s great that the paper provides comparison with image-based MLLMs by extending them to videos and using an LLM to obtain video-level answers - As for the multi-frame approach, it appears that MERLIN is the closest baseline approach. And there seems to be some intersection of pre-training data used for Artemis and MERLIN that includes GOT10K, LaSOT, MeViS - But the evaluation of the video-based referring ability is done on the test set of HC-STVG (lines 197-198), and the train set of HC-STVG is included in the pre-training data for Artemis but not for MERLIN - Since MERLIN was not trained or fine-tuned on HC-STVG, it does not seem to be a fair, apples-to-apples comparison that MERLIN be evaluated on the test set of HC-STVG - On top of that, datasets have biases which includes different label spaces, scene and setting (e.g. HC-STVG is collected from movies), annotation gathering (which can also result in different caption distribution, and hence biasing the eval metrics)

 - Ablation - Assuming “w/o” in Table 2 means that there is no RoI selection (not mentioned or defined in the text), which I’m assuming to mean that there is no <track-instruction>. If the above assumption is true, “w/o” sets up a strong baseline (even better than MERLIN) - In any case, it seems that a careful ablation study is missing - (1) w/o <track-instruction> - (2) w/ <track-instruction> but where <region> features are not RoI features but the key-frame features. This is to establish whether the improvement is brought forth by key-frame selection or region-level features specifically 

 - Visualization and sanity check post-training - Do the authors have an example of a video with multiple different regions? Mainly this is to inspect how the model response changes with selection of different regions in the same video, and whether it doesn’t degenerates to the same response? - Less important, but do the authors have an impression on if the video is reversed, does the caption change? 
 - Human evaluation - Lastly, to substantiate the effectiveness of the approach, did the authors think about a human evaluation study on accuracy / relevance of predicted captions to the video-question pair? - This could also be done within your group and with anonymized predictions (meaning the human evaluator doesn’t know what prediction is from what model) ## Minor - A bit more about the architectural details would have been great (at least in supplemental), especially the tokenization process and whether / how start-end tokens for <instruction> were used - Do the authors have some statistics on the object category of <region> in the test set of HC-STVG? - This is mainly to identify what biases the models are dealing with, and whether those biases skew heavily in one direction. For example, "person" may be the majority category - Similarly, any statistics of actions being performed and asked in questions? - Lastly, do the authors have a quantitative breakdown of performance to understand where the model fails?

Questions

See Weaknesses

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

4

Limitations

Yes