Summary
[Tasks] This paper introduces DaTaSeg, a universal multi-dataset multi-task segmentation model that addresses the interconnections between panoptic, semantic, and instance segmentation tasks.
[Methods] DaTaSeg utilizes a shared representation, consisting of mask proposals with class predictions, across all tasks. To handle task discrepancies, the model employs distinct merge operations and post-processing techniques tailored to each task. Additionally, DaTaSeg leverages weak supervision, enabling cost-effective bounding box annotations to enhance the segmentation model. To facilitate knowledge sharing across datasets, DaTaSeg utilizes text embeddings from the same semantic embedding space as classifiers and shares all network parameters among datasets.
[Experiments] The model is trained on ADE semantic, COCO panoptic, and Objects365 detection datasets. DaTaSeg exhibits improved performance across all datasets, particularly for small-scale datasets, achieving a 54.0 mIoU on ADE semantic and a 53.5 PQ on COCO panoptic. Furthermore, DaTaSeg enables weakly-supervised knowledge transfer for ADE panoptic and Objects365 instance segmentation.
[Results] Experimental results indicate that DaTaSeg scales effectively with the number of training datasets and facilitates open-vocabulary segmentation through direct transfer.
[Dataset] Additionally, the authors have annotated an Objects365 instance segmentation set comprising 1,000 images, which will be released as a public benchmark.
Strengths
1. The authors have conducted comprehensive experiments, showcasing the state-of-the-art performance achieved on multiple long-tailed recognition benchmarks. This highlights the robustness and effectiveness of their proposed method.
2. The paper is skillfully organized, making it easy to follow. The logical structure and clear presentation enhance the reader's understanding of the research.
3. Efficient Use of Supervision. While previous works have explored training universal models on multiple datasets and tasks, a notable strength of this paper is the effective utilization of weak bounding box supervision for segmentation. Compared to full mask annotations, weak bounding box supervision is a more cost-effective and practical solution. This approach makes the proposed method more accessible and applicable in real-world scenarios.
Weaknesses
[Technical contributions on multi-task multi-task model.] One of the key contributions of this paper is the joint training of multiple datasets and multiple tasks within a unified framework. However, it should be noted that this aspect has already been explored by [1]. The referenced work employs a single set of parameters pre-trained for Semantic/Instance/Panoptic Segmentation, Referring Segmentation, Image Captioning, and Image-Text Retrieval tasks. Therefore, it might be necessary for the authors to clarify the differences between this work and X-Decoder.
[Technical contributions on text embedding classifier.] The utilization of a text embedding classifier has also been explored in previous works, such as [2]. In this work, an image encoder is trained to encode pixel embeddings, while CLIP text embeddings are employed as the per-pixel classifier. The key ideas are quite similar, although there are some differences on how to better leverage the text embeddings.
[Lack of comparisons with some published works and subpar performance compared to state-of-the-art methods.] The paper primarily compares the model's performance against methods designed or trained solely on a single task, referred to as "Separate" in the paper. However, it is worth noting that DaTaSeg benefits from training on a significantly larger sample size compared to these listed works. Consequently, the observed performance gains over the "Separate" models are not unexpected. Furthermore, there are published works, such as X-Decoder [1], that jointly train models on multiple datasets and tasks, yielding superior performance on many benchmarks compared to the reported results in this paper.
On ADE semantic (mIoU): X-Decoder achieves 58.1 compared to DaTaSeg's 54.0; on COCO Panoptic (PQ): X-Decoder outperforms DaTaSeg with 56.9 versus 53.5. In most benchmarks, DaTaSeg performs worse than X-Decoder.
[1] "Generalized decoding for pixel, image, and language." Zou, Xueyan, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai et al. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15116-15127. 2023.
[2] "Language-driven Semantic Segmentation." Boyi Li and Kilian Q Weinberger and Serge Belongie and Vladlen Koltun and Rene Ranftl. International Conference on Learning Representations. 2022
Questions
Most of my questions are listed in the Weaknesses section, with my main concerns focused on the technical contributions and performance comparisons with some published works. However, I have a couple of minor questions:
1. Will the code and model weights be made available to the public? It would be valuable to have access to these resources for further exploration and replication of the proposed method.
2. Given that this model can be trained with weakly-supervised tasks, I'm curious if further improvements in performance can be achieved by training on a larger quantity of weakly-supervised samples. Are there any trade-offs between the quality and quantity of the samples when it comes to enhancing model performance?
I believe addressing these questions would provide additional insights into the practicality and potential improvements of the proposed method.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Limitations
Yes, the authors adequately addressed the limitations