OW-VISCapTor: Abstractors for Open-World Video Instance Segmentation and Captioning

Summary

The authors propose a new task called Open-World Video Instance Segmentation and Captioning (OW-VISCap), which involves detecting, segmenting, tracking, and describing both seen and unseen objects. To tackle this problem, they introduce two networks: an object abstractor for encoding images at the object level and an object-to-text abstractor for generating captions for objects. In the object abstractor, they use evenly distributed points as input to obtain open-world object queries, in addition to closed-world object queries, to handle unseen objects. The object-to-text abstractor is composed of standard transformer blocks, but it employs masked attention in the cross-attention layer to focus on objects. The proposed method demonstrates superior performance in Open-World Video Instance Segmentation (OW-VIS) and Dense Video Object Captioning (DVOC) compared to existing approaches.

Strengths

- The authors propose a new task and corresponding solution for video understanding. - They provide a thorough analysis of existing methods in the fields of OW-VIS and DVOC. - Overall, the paper is well-organized and easy to read.

Weaknesses

Despite being the first proposed method for this task, the technical contribution of the proposed approach seems weak. Moreover, the evaluation and analysis appear to lack in several areas: 1. **Evaluation on Limited Benchmarks**: The proposed method has been evaluated on only one benchmark each for OW-VIS and DVOC. To verify its open-world capability, additional benchmarks are needed. Other works like OV2Seg [1] and DVIS++ [2] use a variety of datasets, including LV-VIS, YouTube-VIS, and OVIS, in addition to BURST. 2. **Experimental Evidence for Free-Form Captions**: In lines 33-37, the use of discrete class labels in current OW-VIS methods is cited as an issue. However, there is no experimental evidence showing that using free-form captions improves OW-VIS performance. 3. **Performance in Table 5**: The basic VIS performance appears to be lacking. Recent VIS methods [2-5] achieve over 35 AP on the OVIS benchmark with a ResNet-50 backbone. Even considering the open-world setting, the proposed method's performance gap compared to the latest works in a closed-world setting is too large. References [1] Wang, Haochen, et al. "Towards open-vocabulary video instance segmentation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023. [2] Zhang, Tao, et al. "DVIS++: Improved decoupled framework for universal video segmentation." arXiv preprint arXiv:2312.13305 (2023). [3] Heo, Miran, et al. "A generalized framework for video instance segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. [4] Li, Junlong, et al. "TCOVIS: Temporally consistent online video instance segmentation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023. [5] Ying, Kaining, et al. "CTVIS: Consistent training for online video instance segmentation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Questions

Please refer to the Weaknesses section

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

They discuss limitations of the proposed method in the Appendix.

Summary

This paper propose a new task and the corresponding model: detecting, tracking, segmenting, and captioning open-vocabulary objects in a video. The authors proposed a novel online framework that contains an object detector and feature detector (Object abstractor), a Mask2former style segmentation head, and a frozen LLM. The resulting model is able to detect objects both seen and unseen in the training data, and produce captions. Experiments on open-vocabulary video instance segmentation and dense video object captioning show the proposed model outperform the corresponding SOTA or baselines.

Strengths

- This paper proposed an important and missing new task: segment and captioning all objects in videos. I believe this is one of the ultimate understanding task for video, and this paper defined the task, and composed SOTA components as a novel model for the task. - I appreciate the workload the authors put in composing the model for such a complex task. The model designs of using Mask2Former architecture and a frozen LLM makes sense to me. - Experimental results are reasonable. It shows the proposed method performs better than other models on unseen objects, and produce better captions than alternative stage-wise models or concurrent specialist model. The components are ablated in Table 2 and Table 3.

Weaknesses

- From Figure 3, it is unclear how "video" is handled. Do you feed object queries in the previous frame to a new frame, to retain their identity? If so, would the object caption in different frames be different for the same identity? - While this paper propose a new task, the task is evaluated separately on two sub-tasks on different datasets: OW-VIS and Dense VOC, and thus the overall task is never evaluated. I understand curating evaluation data for this (annotation-heavy) task is expensive, but having some data, even if they are small, will help followup works on this direction. Note this is NOT required in the rebuttal. - The layout of the paper can be improved. E.g., Table 1 is in page 6, but is first referred to in the text at page 8. The paper mentions an frozen LLM is used, but only in the very end of the supplementary the authors unveil which LLM: an OPT-2.7B. I think which LLM is an important model information and should be mentioned in the main text. It is also hard to fine the exact training data to train the model.

Questions

Overall this paper proposed a valid and novel model for a new important task, with good comparisons to existing methods. My concerns are mostly on clarification or presentation, and I hope the authors can clarify/ improve them in the rebuttal. My current rating is a weak accept.

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

4

Limitations

Yes.

Summary

This paper introduces a task called “open-world video instance segmentation and captioning”, which combines open-world video instance segmentation (OW-VIS) and video object captioning tasks. To achieve better performance, the authors propose two key components: an object abstractor to identify new objects using a prompt encoding, and an object-to-text abstractor that bridges object queries with a frozen large language model (LLM) to generate captions.

Strengths

1. The object-to-text abstractor represents a strategic integration of visual data processing with LLM, which enhances the richness and accuracy of the generated captions. 2. The paper is well-written and easy to follow.

Weaknesses

**Method 1. The introduced task is the combination of the existing ones (i.e., OW-VIS and video object captioning), which is not fundamental and novelty enough. 2. For the first challenge (Lines 33-37), some existing vision-language models (VLMs) also focus on free-form captions for instance segmentation (i.e., open-vocabulary segmentation) and object-oriented captioning (e.g., [1]). [1] Dense Video Object Captioning from Disjoint Supervision, 2023. 3. The technical novelty of the proposed method seems limited. Specifically, a) the proposed object abstractor is based on the framework of the video instance segmentation model only with a slight modification (i.e., inter-query contrastive loss). b) the proposed object-to-text abstractor is a captioning model but focuses on describing the objects in the input videos. **Experiment 1. The experiments can be divided into two parts: video instance segmentation and object-oriented captioning. For segmentation, it would be advantageous to utilize more widely used validation sets such as YouTube-VIS. For captioning, besides METEOR, incorporating additional standard captioning metrics like CIDEr and ROUGE would enhance the evaluation. 2. The improvements are slight. a) In Table 1, the proposed method does not achieve SoTA results on more than half of the evaluation metrics compared to baselines, which does not convincingly demonstrate its performance. b) In Table 3, why do the DetA (56.1) and AssA (54.0) metric values remain unchanged, regardless of the addition or removal of components in the ablation study?

Questions

Please refer to the Weaknesses.

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

This paper discusses the limitations in Section G of the supplementary.

Thank you for your rebuttal

Thank the authors for the rebuttal. My confusion on object identities is resolved. I thus keep my positive rating. I respectfully disagree with other reviewers' concern on novelty. To me proposing a working system for video object captioning (with segmentation) is a good enough contribution, as this is important and is missing from the market.