Summary
This paper proposes a fairness continual learning approach in semantic segmentation. Derived from the therotical analysis of the effect of class distribution, a new learning paradigm that is consisted of three main objectives: the task-specific objective to handle the catastrophic forgetting problem, the fairness objective that maintains the fairness predictions over classes, and the conditional structural constraint that imposes the consistency of the segmentation predictions. New SOTA results are attained on the standard benchmarks.
Strengths
1. Deriving the learning objective through analysis of the effect of class distribution is novel.
2. Strong performances are achieved on several standard benchmarks.
Weaknesses
- The title is misleading. In my understanding, the open-world environments means lacking of labels. While the proposed method can hardly work in unsupervised setting.
- Some explanation through mathmatical derivation is not rigorous such as eq. 5 and eq. 12 and 13. More explanation either in the main paper or supplimentary material needed. Detail illustration can be found in the questions.
- Some details are not clear. How the fairness objective $L_{class}$ is calculated during training is not clear and lacking a final integrated training objective. Whether there is hyperparameters between losses is not mentioned.
- According to the ablation study, fairness improvement brought by the fairness objective is minor if measured in STD of IoU over all classes. Better measurement of the fairness is needed. For example, variation of the performance on rare classes and dominant classes with and without fairness loss can be compared in a longer learning steps.
Questions
1. Eq. 5 is hard to understand and I think it could not be intuitively achieved.
2. The explanation that the prototypical contrastive clustering loss is an upper bound of the knowledge distillation approach is not rigorous. In eq. (12) , the $D(p_c, f^{t-1})$ cannot directly ignored as a constant number. When $c$ is a new class from $C^t$, the $p_c$ is changing during learning and $D(p_c, f^{t-1})$ will change. Therefore, $O(D(f^t, p_c)$ is not a tight boundary. Minimizing $L_{cluster}$ is not strictly equals to the minimizing of $L_distill$. From another perspective, although prototypical constrative clustering loss don't need two forward pass of the segmentation model, it have to store the prototypical vector $p_c$ using additional memory. How to calculate the $p_c$ is also critical in achieving good performance.
3. What is the final integrated training loss? Is there any balance weights between different losses?
4. What is the result in the first line of DeepLab-V2 and SegFormer in Table 1?
5. Im my understanding, the conditional structural consistency mitigate the prediction gap among neighbour pixels. However, it will also smooth the edge pixels of an instance. According to the ablation study, it also improves the fairness of classes. Could the author give more explanation on it?
6. Minor problems: when writting the related works, it better to use the surname other than the name. For example, Ziwei et al. should be Liu et al in line 93. In eq. 5, the $y_{i,h}$ should be $y_{i,j}$ in the left term?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Limitations
Authors have pointed out the limitations of the proposed method and some possible improvements.