Active Learning for Semantic Segmentation with Multi-class Label Query

Summary

The paper studied active learning for semantic segmentation. Instead of using each pixel as a query for annotation, this paper used a super-pixel as a query. Since each super-pixel may contain multi-class labels, thus a multi-class labeling scheme is used instead of dominant class labeling. However, it introduced the class ambiguity issue in training since it assigns multi-class labels to each super-pixel region. The authors proposed a two-step learning method, effectively utilizing the supervision of multi-class labels. Extensive experiments were conducted to valid the proposed method, where promising results were reported.

Strengths

++ The idea is quite interesting and practical. Using superpixels as active learning query unit is sensible. Assigning multi-class labels to the superpixels is also quite effective with the proposed two-step learning scheme. ++ The paper is well written and easy to follow. I like the figures provided in the paper, they are quite informative. ++ Good performance, clearly performance gain is obtained comparing with that of dominant class labeling. ++ The training scheme includes merged positive loss and prototypical pixel loss, both of which are well designed loss functions. Both are insightful from my perspective.

Weaknesses

-- It would be interesting to see how the two hyperparameters in Eq. (10) affect the performance. -- It would be better if the number of clicks in Fig.4 can start from 0 instead of 100k. Then we can study the extreme case where very limited number of clicks are provided.

Questions

-- In the right figure of Fig.1, for the 1-class case, multi-class labeling and dominant class labeling should produce the same accuracy and cost the same amount of time. Why they are slightly different in this figure ? -- The paper says "we utilize 32 × 32 superpixel regions given by SEEDS". I am wondering how the superpixel generation method and superpixel size affect the performances. This is an important ablation study to conduct. If the superpixel size is the same as the image size, the method turns to be an image level weakly supervised semantic segmentation task. If the superpixel size is 1*1, the method is the same as using a pixel as the query unit.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

N/A

Summary

This manuscript proposes a new issue on active segmentation from multi-class label query. Concretely, the authors are motivated from the shortcomings of existing active learning paradigm, i.e., [1][2] query the whole image pixel annotations, which costs annotation; [3] selects pixels for annotation, which performs with less generalization. To this end, this manuscirpt follows the compromis that actively annotating the uncertain regions with annotating region-level categories and propose a new query design from multi-hot response. To tackle the issues including uncertain region selecting, category ambiguity, and the absence of fine-grained pixel supervision, the manuscript deviced: a region selection strategy (a.k.a., acquisition function), training with class-ambiguity loss functions (stage 1 training), self-train learning (stage 2 training). Then, the authors conducted experiments on various benchmarks of semantic segmentation to show their effectiveness. [1] Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2019. [2] Lin Yang, Yizhe Zhang, Jianxu Chen, Siyuan Zhang, and Danny Z Chen. Suggestive annotation: A deep active learning framework for biomedical image segmentation. In Proc. Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2017. [3] Gyungin Shin, Weidi Xie, and Samuel Albanie. All you need are a few pixels: semantic segmentation with pixelpick. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2021.

Strengths

1. The draft is beautiful and clear to view. The figures arrayed in Sec. 3 indeed assist me a lot in understanding the detailed methdology and pipeline. 2. The idea of annotate region-level omni-class is novel. Intuitively, it makes a compromis between image-level with pixel-level as authors claimed in the manuscript. 3. The experiments show that the proposed method is effective in saving annotation budget and promise segmentation performance.

Weaknesses

1. The manuscripts required double-check in some details. For example, in L28, "Early approaches consider an entire image as a sample and ask for its pixel-wise class labels [48, 60], or select individual pixels and query the oracle for their class labels [46]; they turned out to be suboptimal since the former lacks the diversity of samples [40] and the latter is less 30 budget-efficient as a query provides supervision for only a single pixel." It is obvious that indivdual pixel-annotation lacks diversity, while whole image annotation costs heavily. But the author happened to claim the opposite of these two points. 2. Despite that the authors deviced a novel issue in segmenting from region-level multi-class query, the method presented seems to be A+B+C style to me. Moreover, the composition seems not to be novel. Firstly, learning from multi-labels for pre-defined sample-level (i.e., region in this paper, image in learning from multi-class classification) via merging pixel loss is not novel. Secondly, Prototypical Pixel Loss and learning from pseudo labels are very common treatment in self-train paradigm, as cited by authors. Actually, I cannot grab the novelty of the manuscript in methdology design. If I misunderstood some points, it will be better to highlight the contributions and add necessary discussion with previous works. 3. The comparison experiments on region selection (a.k.a., acquisition function) with other classic strategy, e.g., KNN, Margin, et. al are missing. Sample selection is of vital importance for active learning, while the authors seems to narrow down the importance to this part.

Questions

I have no questions in evaluating the manuscript. Code is clear.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors did not array the limitations. As for the issues proposed in the Sec. Intro., the adopted methods could work intuitively, and experimental results partly support that.

Summary

This work introduces a new active learning framework for semantic segmentation, involving a novel query design that uses a multi-hot vector for class representation with in a region, two novel loss functions for effective multi-class label supervision, and an acquisition function considering class uncertainty and balance in a local region. This approach optimizes label usage, enhancing supervision and efficiency in semantic segmentation tasks.

Strengths

1) The motivation of this work is clear and interesting. In my opinion, the main strength of this paper lies in the proposed query based on multi-class labels. To be specific, the traditional dominant class labeling suffer from more annotation time per click and bad performance. Therefore, this paper proposed a novel multi-class labeling query and customized two loss and acquisition function for multi-class labeling to handle this problem. 2) The quantitative experimental results are impressive. Only 4% clicks can reach 95% mIoU of the fully-supervised model. 3) Most of this paper are easy to understand.

Weaknesses

- In my view, the Merged Positive Loss and Prototypical Pixel Loss proposed in this paper are essentially minor modifications to the Cross Entropy Loss from a technical standpoint. The Acquisition Function proposed in the paper simply introduces a hyper-parameter to the existing Acquisition Function, incorporating the concept of multi-class.

Questions

- In line 150, Why is Dm represented as D minus Ds? Shouldn't Dm include Ds. - In Figure 1, the Annotation Click for Dominant Class Labeling and Multi-class Labeling are 12.12s and 17.13s, respectively. The author should cite relevant references to make this data convincing to the reader. - In Figure 4, we see that all methods based on multi-class labeling perform better than those based on dominant class labeling. In my opinion, this comparison is not quite fair because multi-class labeling inherently provides more information than dominant class labeling, and thus better performance is to be expected.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Please see the weakness.

Summary

This paper proposes an active learning approach for semantic segmentation using multi-class label queries. Different from dominant class labeling methods, where an oracle is asked to select the most dominant class by a single click, this paper instead designs a multi-class labeling approach that asks the oracle for a multi-hot vector that indicates all classes existing in the given region. Experiments on two benchmarks, Cityscapes and PASCAL VOC 2012, demonstrate promising performance with a significant reduction in annotation cost.

Strengths

1. Reducing annotation costs for segmentation tasks is a long-standing and crucial task. This paper considers active learning to select informative regions from training data and asks an oracle to label them on a limited budget, which is a potential research direction to allow large-scale and annotation-efficient training for segmentation tasks. 2. The proposed method is reasonable and technically sound. 3. The experimental results are comprehensive and verify the effectiveness of the proposed method. 4. The overall paper is well-written and easy to follow.

Weaknesses

1. In stage 1, the total training objective comprises three loss terms. While the sensitivity analysis of the balanced terms $\lambda$ is provided in the supplementary material, I am still wondering whether any findings or insights to determine the importance of each loss term. More insights or clarifications are encouraged. 2. In Figure 7(a), why does the gain of label localization (%) from the method w/ prototype decrease when the number of clicks increases? More clarifications are suggested.

Questions

As mentioned in Weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have stated the limitations and provided the future research directions.

Replying to Rebuttal by Authors

After carefully reading the other reviewers' concerns and the author's responses, I would like to thank the author for addressing all my concerns. I would like to keep the original score.