SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning

This paper introduces a solid state-of-the-art baseline for a\nclass-incremental semantic segmentation (CISS) problem. While the recent CISS\nalgorithms utilize variants of the knowledge distillation (KD) technique to\ntackle the problem, they failed to fully address the critical challenges in\nCISS causing the catastrophic forgetting; the semantic drift of the background\nclass and the multi-label prediction issue. To better address these challenges,\nwe propose a new method, dubbed SSUL-M (Semantic Segmentation with Unknown\nLabel with Memory), by carefully combining techniques tailored for semantic\nsegmentation. Specifically, we claim three main contributions. (1) defining\nunknown classes within the background class to help to learn future classes\n(help plasticity), (2) freezing backbone network and past classifiers with\nbinary cross-entropy loss and pseudo-labeling to overcome catastrophic\nforgetting (help stability), and (3) utilizing tiny exemplar memory for the\nfirst time in CISS to improve both plasticity and stability. The extensively\nconducted experiments show the effectiveness of our method, achieving\nsignificantly better performance than the recent state-of-the-art baselines on\nthe standard benchmark datasets. Furthermore, we justify our contributions with\nthorough ablation analyses and discuss different natures of the CISS problem\ncompared to the traditional class-incremental learning targeting\nclassification. The official code is available at\nhttps://github.com/clovaai/SSUL.\n