TMT-VIS: Taxonomy-aware Multi-dataset Joint Training for Video Instance Segmentation

Training on large-scale datasets can boost the performance of video instance segmentation while the annotated datasets for VIS are hard to scale up due to the high labor cost. What we possess are numerous isolated filed-specific datasets, thus, it is appealing to jointly train models across the aggregation of datasets to enhance data volume and diversity. However, due to the heterogeneity in category space, as mask precision increases with the data volume, simply utilizing multiple datasets will dilute the attention of models on different taxonomies. Thus, increasing the data scale and enriching taxonomy space while improving classification precision is important. In this work, we analyze that providing extra taxonomy information can help models concentrate on specific taxonomy, and propose our model named Taxonomy-aware Multi-dataset Joint Training for Video Instance Segmentation (TMT-VIS) to address this vital challenge. Specifically, we design a two-stage taxonomy aggregation module that first compiles taxonomy information from input videos and then aggregates these taxonomy priors into instance queries before the transformer decoder. We conduct extensive experimental evaluations on four popular and challenging benchmarks, including YouTube-VIS 2019, YouTube-VIS 2021, OVIS, and UVO. Our model shows significant improvement over the baseline solutions, and sets new state-of-the-art records on all benchmarks. These appealing and encouraging results demonstrate the effectiveness and generality of our approach. The code is available at https://github.com/rkzheng99/TMT-VIS .

Paper

Similar papers

Peer review

Reviewer k3N85/10 · confidence 4/52023-07-05

Summary

The paper proposes TMT-VIS, combining multiple VIS datasets for a unified VIS model. Specifically, TMT-VIS introduces taxonomy embedding as prompts to make the model aware of taxonomy from different datasets. Experiments on YVIS-2019, YVIS-2021, OVIS, and UVO demonstrate the effectiveness of dataset unification.

Strengths

+ The proposed method is straightforward and easy to follow. The performance under joint training is good. + Unifying multiple datasets into a single model is of great value.

Weaknesses

- The authors claim that "TMT-VIS is the first DETR-style framework that can jointly train multiple video instance segmentation datasets with such a huge improvement." However, previous work UNINEXT [1] also train multiple VIS dataset and achieves a performance gain with a DETR-style framework, which is missed in the related work. - Tab.3 is confusing. The authors are recommended to evaluate all three datasets in this table to show the performance gain under joint training. Furthermore, the authors claim that Mask2Former-VIS meets difficulties in dealing with datasets, decreasing performance under joint training. Does it indicate the ID-I and ID-V in Tab.3? It is worth noticing that TMT-VIS jointly trained on YTVIS and UVO also decreases compared to the model trained on a single dataset. In other words, the proposed TMT-VIS does not overcome performance drop under joint training. It is ambiguous whether the performance gain is caused by the proposed method boosting the performance on a single dataset or alleviating the conflict in multi-dataset training. - The ablation on the size of taxonomic embedding is not sufficient. For example, what's the performance using all embeddings? Does it have negative effects when the topk taxonomic embeddings miss the gt classes? - Please provide more details of the taxonomy-aware matching loss. [1] Universal Instance Perception as Object Discovery and Retrieval

Questions

please see my listed questions in the weakness part.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors have adequately addressed their work's limitations and potential negative societal impact.

Reviewer VJ6M4/10 · confidence 5/52023-07-06

Summary

Due to the lack of large-scale datasets in the VIS task, the authors propose a multi-dataset joint training method. Due to the heterogeneity of the category spaces of different datasets, simply stacking datasets may lead to performance degradation. For this situation, the author designs a two-stage classification aggregation module, which first compiles the classification information of videos from different datasets, and then Aggregating these categorical information translates categorical priors into instance queries for better performance. The two-stage modules include a classification compilation module and a classification injection module. The former compiles classification information, and the latter utilizes TCM classification information to inject guidance for queries. Experiments prove the effectiveness of this method.

Strengths

The main advantage of this paper is that the proposed joint training model for multiple datasets, which uses taxonomy information to mitigate the heterogeneity in the category space of different datasets, has achieved good results and can try to migrate this approach to other fields.

Weaknesses

The paper lacks clear explanations on the description of the module and the loss function. In the experimental comparison, the performance in the ovis dataset is not satisfactory. Furthermore, the generalization ability of this work on different datasets is limited.

Questions

1.In Taxonomy Compilation Module, what is the specific form or dimension of label space like? Is it a set of category names? 2. Taxonomy-aware matching loss, mask loss, and cls loss - what are the specific formulas for these three losses? 3.In Table 2, Can the effectiveness of the proposed method be verified as it has lower performance on OVIS dataset compared to both IDOL and GenVIS methods? 4.In Experiment V in Table 3, when jointly training with the Youtube-VIS 2019 and UVO datasets, the Mask2Former-VIS method exhibits a decrease of 0.2 in AP compared to using only YTVIS, but it shows an improvement of 3.2 in AP50. However, the TMT-VIS method experiences a decrease of 0.3 in AP and 0.5 in AP50. Can we conclude that the model lacks robustness to differences between different datasets?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Please see the weakness.

Reviewer VJ6M2023-08-18

The author made corresponding explanations and experiments on the robustness of the method. Whether the authors found performance fluctuations caused by dataset imbalance in the original work. In response to this problem, the author does not seem to solve the problem positively, but adopts another joint training strategy. Therefore,combining the comments of other reviewers and the author's responses, I still maintain the original score.

Authorsrebuttal2023-08-20

We deeply appreciate the valuable time and selfless efforts you have dedicated to reviewing and commenting our paper. Dataset imbalance can be roughly divided into two types: category imbalance and dataset scale imbalance. While it's true that scale imbalance could potentially impact the joint training of multiple datasets, we want to clarify that this isn't our primary research focus. In our methodology, we employ a widely used weighted sampling strategy to mitigate the imbalance across various VIS datasets, with ablation studies related to specific hyperparameters of this strategy detailed in Table 5 of our initially submitted paper. Our main research focus is on category imbalance (also referred to as class imbalance in some literature). We address this issue using a two-stage taxonomy aggregation module consisting of: the Taxonomy Correlation Module (TCM), designed to uncover potential taxonomy within each video frame by interacting taxonomic embeddings with video features; and the Taxonomy Integration Module (TIM), developed to aggregate modulated taxonomic embeddings that carry the most relevant taxonomy information. Some categories demonstrate marked improvements (as shown in our supplementary material). As depicted in Table 1, even when training categories that only exist in one VIS dataset, our method still manages to enhance its Average Precision (AP) despite significant category imbalance. Table 1. Comparisons between per-category performance of Mask2Former-VIS and TMT-VIS. ‘MDT’ refers to ‘Multiple Datasets Training’, indicating whether the approach is trained on YTVIS, OVIS, and UVO. The '√' in the middle column is used to demonstrate whether the category is contained in corresponding dataset. For example, all datasets have the 'person' category, and thus all datasets have the corresponding '√'. | Categories | Methods | YTVIS | OVIS | UVO | MDT | AP | | ---------- | ------- | :---: | :--: | :--: | :--: | :--: | | Person | M2F | √ | | | | 57.2 | | | TMT-VIS | √ | | | | 57.9 | | | M2F | √ | √ | √ | √ | 59.3 | | | TMT-VIS | √ | √ | √ | √ | 60.7 | | Duck | M2F | √ | | | | 41.6 | | | TMT-VIS | √ | | | | 42.4 | | | M2F | √ | | | √ | 38.3 | | | TMT-VIS | √ | | | √ | 43.9 | | Monkey | M2F | √ | | | | 24.7 | | | TMT-VIS | √ | | | | 26.7 | | | M2F | √ | √ | | √ | 25.6 | | | TMT-VIS | √ | √ | | √ | 29.1 | | Snowboard | M2F | √ | | | | 8.9 | | | TMT-VIS | √ | | | | 11.8 | | | M2F | √ | | √ | √ | 10.0 | | | TMT-VIS | √ | | √ | √ | 14.5 |

Area Chair kcB32023-08-20

Please expand on why if 3/4 other reviews were positive why you still strongly believe the paper should be rejected? What reasons for acceptance do you disagree with from the other positive reviewers?

Authorsrebuttal2023-08-21

Dear Reviewer VJ6M, We express our heartfelt appreciation for the valuable time and selfless dedication you have invested in reviewing our paper. As the deadline for discussion draws near, we would like to inquire if our response adequately addressed your concerns. We are eager to engage in further discussion. Best regards, Authors of paper 11160

Reviewer fxcA6/10 · confidence 4/52023-07-07

Summary

This paper works on multi-dataset training on video instance segmentation. The authors built on top of Mask2Former, and introduced two components to enable the model to work under different label sets, and take advantages of the given label sets. The two components are both ablated in experiments. The overall framework improved Mask2Former on three popular datasets and achieved state-of-the-art performance.

Strengths

- The problem on training a video instance segmentation model on multiple datasets is important. This paper takes a good step in this direction. The overall framework makes sense to me. - The results are strong. The improvements over the Mask2Former and VITA baselines are significant, and are consistent under three datasets and two backbones. - The contributions are well ablated in Table 4 - 7. - The paper is well written and easy to follow.

Weaknesses

- The most important Table, Table 3, is extremely hard to read. The numbers of different rows are on different datasets, making it hard to compare across rows. PLEASE split the columns by datasets. Please report numbers on all datasets, which should be feasible as the model used a CLIP classifier. We can remove AP50/ AP75 if space is a constraint. - Please clarify: does the model require a known vocabulary during inference? Can it test on in-the-wild images (e.g., using a unified vocabulary)? - [Optional] One other interesting aspect of multi-dataset training in VIS is the difference between the framerate/ image size across different datasets. If will be great if the authors can provide some discussion on this (if this is not trivial).

Questions

Overall the paper works on an important task, proposes a valid model, and has great results. My concerns are mostly on presentation and clarity (see weekness). Please clarify them in the rebuttal.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes.

Reviewer wd7m5/10 · confidence 4/52023-07-07

Summary

This article addresses video instance segmentation from a perspective of multi-dataset joint training. The proposed method is based on DETR, while the main contribution is to inject label taxonomy into model training based on CLIP. The model is evaluated on several standard benchmarks.

Strengths

While multi-dataset training is not a new idea in such fields like object detection, the work is a pioneering study (as far as I know) to explore the idea in video instance segmentation. The paper is overall well written and easy to follow in most parts. The experiments are extensive.

Weaknesses

I would not claim these as weaknesses, but suggestions that might help improve the work. First, while it is good to see that the method shows improvements, I would suggest not characterizing the results as "a such huge improvement". On one hand, a 2%-3% improvement is not huge. On the other hand, the improvements just meet the expectation since more data are used for training. In addition to show quantitative results, it is essential to report some other metrics, such as training time, to render a more comprehensive comparison. Second, as discussed in Sec. 2, there are other strategies for handling heterogeneous labels in multi-dataset joint training, and I am curious whether existing strategies can be directly used for video instance segmentation. If yes, how about the performance? Third, in terms of formulation, E_\infty is used in Sec 3.2, but does not appear in Eqs. 2-5. It is important to ensure consistency. Eq. 6 is not properly formulated as well; a same symbol is used for both input and output, i.e., $X_{l-1}$.

Questions

Beyond the questions in [weaknesses], I have one last question: is the model applicable to open-world scenarios? or at least can it generalize to other datasets that not involve in model training?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Authors tried to discuss limitations; however, the discussion provided is rather elementary and does not offer crucial insights into the method. More in-depth analysis should be provided.

Reviewer 5w4z6/10 · confidence 5/52023-07-09

Summary

In this paper, the authors propose a multi-dataset joint training method for the video instance segmentation task. They build on the MaskFormer-VIS model, introducing two additional modules: the Taxonomy Compilation Module and the Taxonomy Injection Module. And the proposed method shows compelling performance.

Strengths

S1. This paper tackles an overlooked problem: multi-dataset joint training for the video instance segmentation task.

Weaknesses

W1. A significant concern I have is with the explicit relevance and motivation of the proposed methodologies for the task of "video" instance segmentation. The paper currently lacks clear justification as to why these particular methods are designed and show impressive performance specifically for this task. It would benefit from a more in-depth exploration and justification of why these methodologies are uniquely suited for "video" instance segmentation. W2. The paper needs an overall improvement in writing quality. The related works section contains several inaccuracies that need rectification. To specify: - L83: The claim that Mask R-CNN [13] and MinVIS [18] use a tracking branch is incorrect. - L89: It is incorrect to label [47, 28] as MOTS works; they are, in fact, MOT methods. - L90: The assertion that IDOL [39] is based on GenVIS [15] is inaccurate. Additionally, there's a misleading terminology used for the proposed module on L296; it should be the Taxonomy Compilation Module (TCM), not the Taxonomy Extraction Module (TEM). W3. The paper seems to possess marginal novelty. In particular, as stated in L116-117, the proposed method leverages taxonomic embedding (as I understand, these are embeddings from the VLM), rather than language embeddings. The distinction of this work compared to previous multi-dataset joint training methods isn't clear. It would be advantageous if the authors could provide a more explicit elaboration on the unique novelty of their work.

Questions

Major concerns are raised in the weakness section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

1 poor

Presentation

1 poor

Contribution

1 poor

Limitations

This paper has discussed its limitations and societal impact in the main paper.

Reviewer wd7m2023-08-16

thanks for the response

Thanks for the detailed rebuttal and new experiments. Most of my concerns have been properly addressed. But I find Table 2-1 to be somewhat perplexing since the dataset numbers and underlying backbones are different. This makes it hard to make a proper comparison. In addition, I read other reviewers' comments and would like to discuss more following Reviewer 5w4z's `why video?` question. On one hand, the model shows favorable performance in the new `image` setting, i.e., a gain of +4% is undoubtedly promising. But does this outcome imply that the method's uniqueness is not exclusively tied to `video`? On the other hand, the newly added experiments coutine to rely on YTVIS-19. I am actually curious about results on datasets exclusively comprised of images like COCO and LVIS. While I don't anticipate fresh results at current stage, the question seems to be remain unanswered and I expect authors give more insights.

Authorsrebuttal2023-08-20

We are sincerely grateful to you for the precious time and selfless efforts you have devoted to reviewing and commenting our paper. Our research is primarily anchored on Video Instance Segmentation (VIS), not image instance segmentation, and we are willing to accept all types of results—be it advancements or drawbacks—stemming from jointly training image datasets. We are exploring the usage of joint training methods in VIS training with a view to combine existing standalone and specific VIS datasets into a more exhaustive resource. Our motivation stems from the notable size difference between current image instance segmentation datasets and VIS datasets. As an example, LVIS hosts 160k images together with 2M instance mask annotations, thereby providing a richer variety of scenarios compared to VIS datasets. Conversely, VIS datasets are typically smaller and the resources available to us are largely fragmented, field-specific datasets rather than larger, established ones. Given that there's no prevalent joint training strategy for VIS and a lack of widely recognized large-scale VIS datasets, our proposition is to jointly train the available VIS datasets to enhance overall VIS performance. Table 1 Key statistics of Image Instance Segmentation datasets | | COCO | LVIS | | ---------- | ---- | ---- | | Images | 164K | 160K | | Categories | 80 | 1203 | | Masks | 1.2M | 2M | Table 2. Key statistics of multiple VIS datasets. | | YT19 | YT21 | OVIS | UVO | | ----------------- | ---- | ---- | ---- | ------ | | Videos | 2883 | 3859 | 901 | 11228 | | Categories | 40 | 40 | 25 | 81 | | Instances | 4883 | 8171 | 5223 | 104898 | | Masks | 131k | 232k | 296k | 593k | | Masks per Frame | 1.7 | 2 | 4.7 | 12.3 | | Objects per Video | 1.6 | 2.1 | 5.8 | 9.3 |

Reviewer k3N82023-08-17

I would like to thank the authors for the detailed rebuttal. Most of my concerns are addressed. The proposed method shows non-trivial performance in unifying multiple video segmentation datasets into a single model. However, I agree with Revier 5w4z's concern about the uniqueness of video tasks. For instance, UNINEXT unifies not only VOS but also other detection, segmentation, and reference comprehension tasks on both video and image. It is somehow unclear the motivation for how the proposed method target boosting VOS instead of image segmentation, or the authors just follow the trend of multi-dataset training. I expect the authors to provide deep insights into how the proposed method effectively unifies the multi-dataset VOS and boosts the performance of each dataset.

Authorsrebuttal2023-08-20

We extend our deepest appreciation for the invaluable time and selfless efforts you have dedicated to reviewing and providing insightful comments on our paper. In fact, our research focus remains on VIS task rather than image instance segmentation, and we are open for the results (no matter improving or degrading) on jointly training image datasets. Our research motivation is that current image instance segmentation datasets are significantly larger than VIS datasets. LVIS, for example, has 160k images and 2M instance mask annotations. The scenarios contained in these images are thus more varied than VIS datasets. Such image dataset is so large and dominant that jointly training multiple image instance seg datasets become less exciting and influential. On the other hand, VIS datasets are smaller in scale, and what we possess are numerous isolated field-specific datasets, rather than dominant scale datasets. Due to this fact, combining these varied VIS datasets and training them jointly become useful and meaningful for VIS research. Also, in our early experiments we noticed that adapting pretrained MAE-based weights (ImageMAE \& VideoMAE ) to Mask2Former structure is unsuccessful (please refer to Table 1). Based upon these, we are motivated to research on applying joint training methods in VIS training, hoping to unify these separate specific datasets. A major challenge in multiple-dataset joint training is the heterogeneity of multiple datasets’ category space. As a result of the difference in category space, though mask precision increases with the data volume, dataset biases might hinder models from generalization: simply utilizing multiple datasets will dilute the attention of models on different categories. Therefore, increasing the data scale and enriching label space while improving classification precision become a huge challenge for researchers. This phenomenon was shown in our ablation studies. Our method, adjusts the language embeddings via a taxonomy compilation module which consists of a spatio-temporal adapter and a FFN network, as well as of a taxonomy injection module which fuses the taxonomy information into video queries with a multi-level cross-attention and self-attention layers. In the taxonomy compilation module, taxonomy embeddings are designed to interact with video features in order to unearth the potential taxonomy contained in each of the video frames. After this, we calculated the dot product between the different modulated taxonomic embeddings to predict the most relevant taxonomy in the given video. By compiling and injecting the possible taxonomic information to queries as guidance, queries tend to converge to the desired instances and desired categories faster and finally come up with better precision. Also, when we constrain the most relevant taxonomy of input video, we no longer have to worry about the full heterogeneous category space. As depicted in Table 2, even when training categories that only exist in one VIS dataset, our method still manages to enhance its Average Precision (AP). Table 1. Experiments on Adapting ImageMAE \& VideoMAE to VIS tasks | Method | Backbone | AP | | ------------ | -------- | ---- | | M2F | SWIN-B | 59.5 | | ImageMAE+M2F | VIT-B | 53.3 | | VideoMAE+M2F | VIT-B | 27.1 | Table 2. Comparisons between per-category performance of Mask2Former-VIS and TMT-VIS. ‘MDT’ refers to ‘Multiple Datasets Training’, indicating whether the approach is trained on YTVIS, OVIS, and UVO. The '√' in the middle column is used to demonstrate whether the category is contained in corresponding dataset. For example, all datasets have the 'person' category, and thus all datasets have the corresponding '√'. | Categories | Methods | YTVIS | OVIS | UVO | MDT | AP | | ---------- | ------- | :---: | :--: | :--: | :--: | :--: | | Person | M2F | √ | | | | 57.2 | | | TMT-VIS | √ | | | | 57.9 | | | M2F | √ | √ | √ | √ | 59.3 | | | TMT-VIS | √ | √ | √ | √ | 60.7 | | Duck | M2F | √ | | | | 41.6 | | | TMT-VIS | √ | | | | 42.4 | | | M2F | √ | | | √ | 38.3 | | | TMT-VIS | √ | | | √ | 43.9 | | Monkey | M2F | √ | | | | 24.7 | | | TMT-VIS | √ | | | | 26.7 | | | M2F | √ | √ | | √ | 25.6 | | | TMT-VIS | √ | √ | | √ | 29.1 | | Snowboard | M2F | √ | | | | 8.9 | | | TMT-VIS | √ | | | | 11.8 | | | M2F | √ | | √ | √ | 10.0 | | | TMT-VIS | √ | | √ | √ | 14.5 |

Reviewer fxcA2023-08-18

Thank you for the rebuttal

Thank you for the rebuttal. My concerns are all well addressed in the rebuttal. I keep my original positive rating.

Authorsrebuttal2023-08-20

Dear Reviewer 5w4z, We are sincerely grateful to you for the precious time and selfless efforts you have devoted to reviewing our paper. Since the deadline of discussion is approaching, we would like to inquire whether our response has addressed your concerns and if you have the time to provide further feedback on our rebuttal. We are more than willing to engage in further discussion. Best regards, Authors of paper 11160

Reviewer 5w4z2023-08-20

Thank you for the authors' response. I have carefully reviewed the feedback from other reviewers and the authors' comments on them. In particular, both Reviewer k3N8 and Reviewer wd7m shared my initial question regarding the choice of "why video?". I've read the authors' explanations on this matter. I understand that video instance segmentation datasets tend to be smaller than image datasets, and that joint training for a large vocabulary presents its own set of challenges. My primary concern, however, centers on the unique benefits the proposed method offers for video. As far as I know, M2F-VIS uses 2 frames from a single video during the VIS task training. Have the authors followed this implementation setting? Authors mentioned integrating a 'spatio-temporal adapter to generate video-specific modulated taxonomic embeddings.' I wonder if using just two frames can effectively capture video-oriented knowledge for the model and assess its efficacy.

Authorsrebuttal2023-08-21

We express our sincere gratitude to you for generously dedicating your valuable time and selfless efforts to thoroughly review and provide insightful comments on our paper. As shown in previous ablations in submitted paper and rebuttal, we show that our strategy is plug-and-use to popular VIS methods across VIS datasets. In experiments on M2F-VIS, we follow the original settings to set the input video frames per video to 2. While in experiments based upon VITA, when input video frames per video is set to 6, the overall performance becomes SOTA with our design. The effectiveness of capturing video-oriented knowledge with different input frames in the design of TCM is a problem worth discussion, but is not the main focus of our current paper. In M2F-VIS setting, we previously experimented that M2F-VIS tends to have a drop in performance when input frame number grows larger than 2 (in Table 1). This indicates that 2 frames are a basic module for M2F-VIS to build up temporal correlations. Even though with more input frames comes more features, the complexity of modeling instance trajectories, on the other hand, hinders the performance. In such settings, 2 is the most effective choice of parameter. As for capturing video-oriented knowledge with more input frames, we haven't fully experimented on this, which is not the main focus of our work. Table 1. M2F-VIS performance on YTVIS-19 on R-50 backbone | | 1 | 2 | 3 | 4 | 6 | | ------- | :--: | :--: | :--: | :--: | :--: | | M2F-VIS | 45.1 | 46.4 | 46.0 | 45.7 | 45.6 |

Reviewer 5w4z2023-08-21

Thank you for the comments! I concur that the current M2F-VIS design isn't ideally suited for capturing temporal relationships, and its end-to-end nature might substantially increase resource demands. However, with the use of multiple frames as input and observing the results from VITA, which showcases SOTA performance, my reservations have been alleviated. As a result, I'd like to amend my rating to "weak accept". I hope the authors delve deeper into these concerns and provide further clarifications in their revised paper.

Reviewer 5w4z2023-08-21

Also, taking into account the responses provided by the authors above, I believe that ablation studies using VIS methods other than M2F-VIS would be greatly beneficial for readers.

Authorsrebuttal2023-08-21

Dear Reviewer 5w4z, We greatly appreciate your acknowledgement of our rebuttal and your appreciation of our work. We sincerely thank you for your valuable feedback and support. In accordance with your suggestions, we will diligently refine the paper and address any confusing sections. Your constructive feedback will contribute to enhancing the overall quality of our work. Once again, thank you for your invaluable contribution. Best regards, Authors of paper 11160.

Reviewer 5w4z2023-08-21

Finally, I'd like to draw attention to the related work section. For expert readers, such as reviewers, having misleading content in this section can significantly diminish the credibility of the paper. While the authors have acknowledged and made some revisions, I urge them to further review and refine this section carefully. It's essential for the authors to take responsibility for their content, especially since many subsequent studies reference the related work sections of top-conference papers.

Area Chair kcB32023-08-20

Response Required

Dear Reviewer 5w4z, Given you are in the minority with a reject it is critical that you read the rebuttal and inquire about any outstanding issues you feel are still not addressed. Thank you, AC

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC