Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations

Deep neural networks suffer from the major limitation of catastrophic\nforgetting old tasks when learning new ones. In this paper we focus on class\nincremental continual learning in semantic segmentation, where new categories\nare made available over time while previous training data is not retained. The\nproposed continual learning scheme shapes the latent space to reduce forgetting\nwhilst improving the recognition of novel classes. Our framework is driven by\nthree novel components which we also combine on top of existing techniques\neffortlessly. First, prototypes matching enforces latent space consistency on\nold classes, constraining the encoder to produce similar latent representation\nfor previously seen classes in the subsequent steps. Second, features\nsparsification allows to make room in the latent space to accommodate novel\nclasses. Finally, contrastive learning is employed to cluster features\naccording to their semantics while tearing apart those of different classes.\nExtensive evaluation on the Pascal VOC2012 and ADE20K datasets demonstrates the\neffectiveness of our approach, significantly outperforming state-of-the-art\nmethods.\n