AUCSeg: AUC-oriented Pixel-level Long-tail Semantic Segmentation

Summary

The authors extend AUC optimization techniques to pixel-level long-tail semantic segmentation and propose a general pixel-level AUC loss function. They decompose the loss function into inner-image and inter-image terms to decouple the interdependency, and calculate bounds to theoretically prove the effectiveness of this loss. Additionally, they design a Tail-Classes Memory Bank to reduce memory demand. Qualitative and quantitative comparisons have been made on three benchmark datasets, showing the efficiency of their method.

Strengths

1. The paper is clearly organized and well written. 2. The general idea of the paper is clear, and the motivation is interesting. 3. The authors provide detailed theoretical proofs.

Weaknesses

1. In the tail-class memory bank module, how do you ensure that pasted tail pixels do not completely cover any category? 2. As shown in Fig 3(b), authors argue that because images contain multiple labels, it is impossible to use stratified sampling techniques to ensure each mini-batch contains at least one sample from each class. However, I believe this view is incorrect; we can group images by category, different categories may include identical images due to multi-label nature. Then we can use stratified sampling technique for each category so that although it might sample identical images (the same sample being sampled twice as both head and tail), it still ensures each mini-batch contains at least one sample from every class, and small batch sizes can also achieve this goal. Therefore I think there’s no need for designing a Tail-Class Memory Bank. 3. How does Tail-Class Memory Bank store data? Does it directly use ground truth or predicted results? If using ground truth directly why design Store Branch? You could directly build memory bank using ground truth. If using predictions then why not just use ground truths? 4. Pasting tail-class pixels onto original image might occlude original image classes; would this behavior affect performance? 5. This method mainly relies on SegNeXt; while sometimes environmental factors prevent exact replication of other works' results. However, SegNeXt provides training logs & pre-trained weights, why did your reproduction show significant performance gaps compared with SegNeXt's original work (e.g., nearly 4% gap on ADE20K)? 6. In Table 1, this method shows significant improvement only on ADE20K but limited gains across two other datasets suggesting poor real-world generalizability inconsistent with theoretical conclusions. 7. Currently effective methods trained over small datasets often fail scaling up large-scale models. Hence the author should further provide the results with large-scale pretrained models like SAM/CLIP etc., demonstrating broader applicability/generalization potentiality thereof.

Questions

1. I am confused about Tail-Class Memory Bank design, please refer weaknesses section above explaining them accordingly. 2. Authors should clarify certain experimental outcomes referring weakness points mentioned earlier. 3. NeurIPS Paper Checklist lacks honest responses e.g., code provision/error bars reporting/new assets do not be contained, but all marked “YES”.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors have addressed the limitation of this method. This method might slightly impair the performance of head classes due to an increased focus on tail classes.

Summary

This paper explores AUC optimization methods for pixel-level long-tail semantic segmentation, addressing complex dependencies and space complexity challenges. The authors propose a novel pixel-level AUC loss function and conduct a dependency-graph-based theoretical analysis to enhance generalization. They also introduce a Tail-Classes Memory Bank (T-Memory Bank) to manage memory demands, and experiments confirm the effectiveness of their AUCSeg method.

Strengths

1. This paper presents a novel perspective by using AUC as the loss function. 2. The paper has performed a tail class cache to boost the performance and control the memory demand.

Weaknesses

Please refer to the questions.

Questions

1. For a long-tail semantic segmentation problem, which is also a pixel-wise classification task, I am uncertain why the authors chose this perspective. From the standpoint of the loss function, it emphasizes the contrast between the current class and other classes. Would a contrastive classification approach achieve similar results? 2. The significance of Proposition 1 is not very clear. What is the meaning of the $\Omega$? My understanding is that for a tail class, it is naturally difficult to randomly sample such a sample due to the scarcity. Therefore, re-sampling techniques are needed, which seems quite intuitive. What additional information is this proposition supposed to convey? 3. The logic of using a memory bank in this paper seems weired. The authors argue that it is difficult to randomly sample a tail sample, hence the need for a memory bank. Most long-tail based work considers batch re-sampling a very conventional approach. However, this paper's method enhances the diversity of tail samples, thereby improving the model's learning ability for tail samples. Perhaps rephrasing this aspect would be more effective.

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

Yes

Summary

This paper investigates AUC optimization within the context of pixel-level long-tail semantic segmentation (PLSS), a complex task due to intricate loss term coupling and extensive memory requirements. Initially, the authors demonstrate the potential of AUC for PLSS from a theoretical perspective by addressing the two-layer coupling issue across loss terms. Subsequently, they propose a novel Tail-Classes Memory Bank (T-Memory Bank) to manage the significant memory demands of AUC-oriented PLSS. Finally, comprehensive experiments show the effectiveness of the proposed method.

Strengths

1. This paper addresses a compelling problem with clear motivations and significant contributions, offering insightful ideas for both the AUC and segmentation communities. 2. Developing a theoretically grounded loss from an AUC perspective is novel, and the proposed T-memory bank effectively mitigates the memory burden associated with pixel-level AUC optimization. 3. The performance of the proposed AUCSeg seems promising.

Weaknesses

Overall, I believe this is a qualified paper for publication after addressing the following minor concerns: 1. The notations in this paper should be carefully defined. Some key symbols are used repeatedly, such as $N$ representing both batch size (In Alg.1) and sample size (In Thm.1). 2. This paper only examines the square AUC surrogate loss without considering two other popular surrogate losses for AUC optimizations (i.e., hinge and exponential losses). It is essential to provide empirical verification by applying the ignored losses to AUCSeg. 3. In Alg.1, the batch size scale will also impact the final performance of long-tailed classifications. The authors are recommended to conduct an ablation study on this aspect.

Questions

Please refer to the weakness part.

Rating

8

Confidence

5

Soundness

4

Presentation

3

Contribution

4

Limitations

None

Summary

This paper introduces AUC optimization into the domain of long-tailed semantic segmentation. Specifically, the authors developed a pixel-level AUC loss function tailored for long-tailed semantic segmentation tasks and introduced a tail-class memory bank to address the memory demands. Additionally, the authors utilized Rademacher complexity to provide a generalization bound for AUCSeg and theoretically analyzed the potential for AUCSeg to generalize to unseen data. The experimental evaluation in this paper assessed the performance of the proposed strategy on various datasets. Generally speaking, this paper is relatively detailed and comprehensive.

Strengths

- It is novel to introduce auc optimization in long-tailed semantic segmentation and give a detailed analysis over the feasibility of this method. - The proposed method is validated on different datasets(ADE20k, CityScapes, COCO etc.) and has been proven to be effective in tail classes.

Weaknesses

There are a few concerns about this paper: - On COCO, the performance improvement of Tail Class seems to be limited. Could the author explain the reason? - Fig. 6(a) shows the memory size reduction of the tail-class memory bank. The results show that as the memory size increases, the performance improvement decreases. What is the reason for this?

Questions

I thank the author for their detailed work. Nevertheless, there are a few questions I wonder and I hope the author could respond. - For Tail Class Memory Bank, what would happen if we use a pixel-level memory bank? For example, prototype or similar technology, would this lead to significant performance degradation? - For the memory bank update problem, is there a more appropriate selection method instead of random replacement?

Rating

5

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

yes