Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning

Few-shot class incremental learning (FSCIL) portrays the problem of learning\nnew concepts gradually, where only a few examples per concept are available to\nthe learner. Due to the limited number of examples for training, the techniques\ndeveloped for standard incremental learning cannot be applied verbatim to\nFSCIL. In this work, we introduce a distillation algorithm to address the\nproblem of FSCIL and propose to make use of semantic information during\ntraining. To this end, we make use of word embeddings as semantic information\nwhich is cheap to obtain and which facilitate the distillation process.\nFurthermore, we propose a method based on an attention mechanism on multiple\nparallel embeddings of visual data to align visual and semantic vectors, which\nreduces issues related to catastrophic forgetting. Via experiments on\nMiniImageNet, CUB200, and CIFAR100 dataset, we establish new state-of-the-art\nresults by outperforming existing approaches.\n