Summary
[Task] In this work, authors introduc SEEM, a promptable and interactive model designed for comprehensive image segmentation. SEEM aims to segment all objects in an image simultaneously, addressing various segmentation tasks. The key contribution of SEEM is its novel decoding mechanism, which allows for diverse prompting and behavior similar to large language models (LLMs).
[Method] SEEM is built with four key objectives: versatility, compositionality, interactivity, and semantic-awareness. The model incorporates a visual prompt that unifies different spatial queries such as points, boxes, scribbles, and masks, enabling generalization to different referring images. It also learns a joint visual-semantic space for dynamic prompt composition in different segmentation tasks. Additionally, SEEM incorporate learnable memory prompts in the decoder to retain segmentation history and utilizes mask-guided cross-attention from the decoder to image features. Moreover, a text encoder is employed to encode text queries and mask labels into the same semantic space, facilitating open-vocabulary segmentation.
[Experiments] Empirical studies validate the effectiveness of SEEM across diverse segmentation tasks. The model achieves competitive performance in interactive segmentation, semantic segmentation, referring segmentation, and video object segmentation across nine datasets, requiring only minimal 1/100 supervision. SEEM demonstrates remarkable generalization capability to novel prompts or their combinations, positioning it as a versatile and universal image segmentation interface.
Strengths
[highlight the trend towards more flexible segmentation models] In this work, the authors address the need for a universal segmentation interface capable of handling different types of human prompts and addressing various segmentation tasks. They highlight the trend towards more flexible segmentation models, including open-vocabulary segmentation, referring segmentation, and interactive segmentation. Inspired by the success of Large Language Models (LLMs) as universal interaction interfaces for language tasks, the authors propose SEEM, a promptable and interactive model for segmenting everything everywhere all at once in an image.
[technical contribution] The proposed approach, SEEM, introduces a novel prompting scheme in the mask decoder with four key properties: versatility, compositionality, interactivity, and semantic-awareness. By encoding different types of prompts (points, masks, text, boxes, and referred regions) into a joint visual-semantic space, SEEM achieves strong compositionality and can handle any combination of input prompts. Memory prompts are introduced to retain previous segmentation information and enable interactivity. The model also provides open-set semantic labels for output segmentation.
[tasks and results] The model is trained on diverse segmentation tasks to learn how to handle different prompts, align visual and text prompts, and promote their synergy through cross-attention. The single pre-trained SEEM model achieves competitive performance across all segmentation tasks, leveraging the joint visual-semantic space for prompt combination and zero-shot adaptation. In addition to its strong generalization capability, SEEM is efficient for interactive segmentation compared to other methods. Overall, SEEM offers a segmentation interface with a single pre-trained model that can handle all types of prompts, segment every object with semantics, and cover every pixel in the image.
Weaknesses
[Model performance] The model performance of SEEM on certain benchmarks, including ADE20K for open-vocabulary panoptic segmentation and DAVIS for video instance segmentation, is often lower, and in some cases, significantly lower than the baselines X-Decoder, ODISE, and UNINEXT. It is worth noting that SEEM's framework follows a similar approach to X-Decoder.
While the concept of a universal and interactive segmentation interface is intriguing, it appears that the model performance on open-vocabulary benchmarks is compromised in SEEM. It would be beneficial for the authors to provide an explanation for this discrepancy.
[Unfair comparisons] In Table 2, there are some comparisons that seem unfair for two reasons. Firstly, the evaluation of SEEM on COCO, which provides instance-level annotations, aligns better with SEEM's training pipeline that includes instance-level annotations. However, this evaluation method puts SAM at a disadvantage since it is trained to respect hierarchical segmentation, including whole-part-subpart relationships. Secondly, SEEM is already trained on in-domain data from COCO's training set, while SAM is pretrained on out-of-domain data. This discrepancy in training data can impact the performance comparison between the two models. To ensure fair comparisons, it would be beneficial to evaluate both models using evaluation protocols that align with their respective training pipelines and take into account the domain of the training data.
Questions
[Handling dataset imbalance] Since SEEM is trained on multiple datasets, there might be an imbalance in the number of samples per dataset. I'm curious to know if adjusting the frequency of training samples from each dataset has any impact on the model's performance. Can the authors shed some light on whether they have explored techniques such as data augmentation or sample weighting to address this issue? Additionally, it would be interesting to understand if adjusting the training sample frequency has any effect on mitigating the impact of dataset imbalance and improving the overall model performance.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Limitations
Yes, the authors adequately addressed the limitations