Few-Shot Class-Incremental Learning via Training-Free Prototype Calibration

Summary

The paper proposes a strategy called Training-frEE calibratioN (TEEN) for Few-Shot Class-Incremental Learning (FSCIL) scenario to enhance the discriminability of new classes by fusing the new prototypes with weighted base prototypes. TEEN demonstrates remarkable performance and consistent improvements over baseline methods in the few-shot learning scenario.

Strengths

The paper addresses the Few-Shot Class-Incremental Learning (FSCIL) scenario, which is a challenging and important problem in real-world scenarios. The proposed strategy, TEEN, is simple yet effective and demonstrates remarkable performance and consistent improvements over baseline methods in the few-shot learning scenarios.

Weaknesses

The paper demonstrates significant improvement on new classes. However, based on the experimental results, it appears that the False Negative ratio, which involves classifying base instances into incorrect classes, may increase. It would be beneficial to address how this problem is handled and how the performance can be balanced between base classes and new classes. Additional analysis on this topic would be appreciated. As stated by the authors, pre-training on a dataset that is independent of the subsequent data distribution would be advantageous. The paper lacks a comparison of the proposed method with state-of-the-art methods in other few-shot learning scenarios, such as few-shot domain adaptation or few-shot semi-supervised learning. It would be beneficial to include more analysis and comparisons with these methods.

Questions

See the weakness section for more details

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See the weakness section for more details

Summary

This paper tackles the problem of few-shot class incremental learning based on prototypical network. Motivated by the observation that novel classes are easily misclassified as base classes, the authors propose a prototype calibration strategy. The calibrated prototype is a weighted sum of prototype computed from novel class support set and base classes semantically similar with the novel class. Experiments on miniImageNet, CUB and CIFAR demonstrate the superiority of the proposed method over previous ones.

Strengths

1. The paper is well motivated that the inferior performance of prototypical network results from the phenomenon that novel classes are easily misclassified as base classes. The phenomenon is well-studied via experiments. 2. The paper is well written and easy to follow. 3. The authors propose a simple but effective calibration strategy to improve the performance.

Weaknesses

1. The paper introduces two hyper parameters which need to be exhaustively searched for every dataset. 2. There lacks cross-dataset experiments to show the effectiveness of the proposed method when transferring knowledge from one dataset to another.

Questions

The semantic similarity based calibration is a straightforward and commonly used method used in the classification area. There may lack technical contribution to the community.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The limitation of cross-dataset evaluation is discussed.

Summary

The authors work on the Few-Shot Class-Incremental Learning (FSCIL) scenario and propose the Training-frEE calibratioN (TEEN) strategy. This strategy enhances the discriminability of new classes by fusing the new prototypes with base prototypes. This approach is different from previous methods, which either introduce extra learnable components or rely on a frozen feature extractor.

Strengths

1. The authors observed that the feature extractor, although only trained on base classes, can surprisingly represent the semantic similarity between the base and unseen new classes. Accordingly, they proposed a solution. 2. The paper is easy to follow, with clear experimental details that aid reproducibility. The motivations are also presented clearly.

Weaknesses

1. The issue of misclassification has already been observed and studied in previous few-shot learning works. Therefore, the poor performance of new classes is not surprising in the task of Few-Shot Class-Incremental Learning (FSCIL). 2. Forming connections across samples from base and novel classes is not a new concept. For instance, FADI[1] discovered that a novel class may implicitly leverage the knowledge of multiple base classes to construct its feature space. It then builds a discriminative feature space for each novel class via association and discrimination steps. [1] Few-Shot Object Detection via Association and Discrimination, Yuhang Cao et al., NeurIPS 2021.

Questions

I can understand that the prototype of a novel class is biased due to the lack of samples, and it can be easily misclassified as a base class. Intuitively, if the novel class is prone to be classified as the most similar base class, interpolation among such features could exacerbate this issue. But why is the use of weighted base prototypes to calibrate novel ones beneficial in enhancing the discriminability of such a prototype while reducing the discriminability of base prototypes (lines 217-218, 238-239)?. And could you please clarify the definition of discriminability?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

As mentioned in the weaknesses section, the proposed idea is not novel at all. It unfortunately is just a tiny incremental contribution which is rather insufficient for a publication in top tier conference such as NeurIPS.

Summary

This paper presents a novel training-free calibration approach for few-shot class incremental learning. The authors make an observation that one main problem with FSCIL is that data of new classes can be easily mis-classified as base session classes. By utilizing the well-calibrated embeddings of base session classes to help embeedings of the new classes, the author improve the performance of FSCIL.

Strengths

1. The observation of new classes being confused as old classes is somewhat novel. 2. The proposed classifier-calibration method is interesting and novel. 3. The proposed method is simple yet effective. 4. Detailed ablation studies have been performed on the introduced hyper-parameters.

Weaknesses

1. One of the reviewer's concern is in terms of the robustness of the method to hyper-parameter \alpha and \tau. Are the optimal hyper-parameters the same for different datasets or different incrementing procedures (i.e., how many classes per incremental session)? 2. In addition, the author should provide more detailed justification for the design of the method. Why use this simple linear interpolation for calibration? Are there any empirical observations or theory to support this design?

Questions

1. I suggest the author provide more ablation study on robustness of hyper-parameter on more dataset. 2. I suggest the author provide more justification on the designing of the method (i.e., why this simple linear interpolation form). Such justification can be empirical visualizations, theoretical analysis, etc.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors have adequately addressed the limitations.

Summary

The paper closely looks at problems in prototypical networks and methods in the context of few shot class incremental learning. The authors observe semantic similarity between new and base prototypes along with new classes being misclassified regularly as semantically similar base classes. Towards this they use a training free calibration strategy to guide the new ill calibrated prototypes using the base prototypes. Essentially bridging the lack of training data and suppressing bias.

Strengths

- The major strength of the paper lies in the simplicity of the method. Instead of compromising the base class performance they approach the problem from a new class orientation. Doing so relieves the method from the convoluted approaches of feature space reservation[1], meta learning schemes[2] or self supervision[3]. Instead the methods strength lies in utilising the uncompromised power of the base prototypes. - The fact that almost all evaluation metrics one way or the other focus on the novel class performance and not the less interpretable average accuracy and performance decay rate makes the paper even more attractive. This shows commitment towards pushing the frontier in aspects previously underexplored and facilitating a fair and thorough comparison. - The work is constructed clearly and simply. It seems obvious that the authors followed a train of thought. They end up convincing the reader on why each module is essential. It generally reads very smoothly, hopping from hypothesis to observations to results and repeats. For example section 3 which tries to “Understand the reason for poor performance in new classes” leads smoothly into the observations for that section which motivates the section on calibration (section 4) and this section takes a similar hypothesis-observation-result route. - The fact that almost all evaluation metrics one way or the other focus on the novel class performance and not the less interpretable average accuracy and performance decay rate makes the paper even more attractive. This shows commitment towards pushing the frontier in aspects previously underexplored and facilitating a fair and thorough comparison.

Weaknesses

- Session 4 uses a variety of terminology. Using a consistent term for each prototype could potentially read better. (For example ill-calibrated new prototypes, new prototypes, biased prototypes) - In relation to prior work it seems only Section 1 is truly dedicated to the literature. And even then it seems the prior works that this paper branches out from are not discussed in appropriate detail. Prior methods from the 3 other papers from Table 1 and Table 2 are not detailed in any preceding or subsequent section. Their would be reason to do so as the method directly criticises (line 86-87 or line 190-191) prior works. Prefacing in better detail how previous methods mitigate biases could be essential in understanding the state of the literature and motivate TEEN. - [4] show in “Simpleshot” that L2 normalisation of prototypes improves performance. This is also confirmed by [5] in their nearest neighbour method for few shot learning. The authors overlooked this crucial feature transformation which is commonly conducted in previous studies on Non Parametric methods for FSCIL for example the NoNPC method by Oh et al. [6] This omission somewhat limits the comprehensiveness of their findings despite the improved results given the prevalnece of the method. - [7], [8] are missing comparisons. Therefore, the tables that compare the method to the current sota would should less prominent results. Minor Remarks and Typos: - 213: “rely” should be “relies” - 2: “scarce” - The use of tau parameter in equation (5) might be better understood if shifted to equation 6 as S_{b,n} is a scalar and in any case it seems more common that a temperature scaling parameter appears in the softmax. Would recommend to make that alteration. - 73: “neglect” should be “negligence” [1] Zhou, Da-Wei, et al. "Forward compatible few-shot class- incremental learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022. [2] Zhang, Chi, et al. "Few-shot incremental learning with continually evolved classifiers." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021. [3] Ahmad, Touqeer, et al. "Few-shot class incremental learning leveraging self-supervised features." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. [4] Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens van der Maaten. Simpleshot: Revisiting nearest-neighbor classification for few- shot learning. arXiv preprint arXiv:1911.04623, 2019. [5] Wang, Guangpeng, and Yongxiong Wang. "Self-attention network for few-shot learning based on nearest-neighbor algorithm." Machine Vision and Applications 34.2 (2023): 28. [6] Oh, J., & Yun, S.-Y. (2022). Demystifying the Base and Novel Performances for Few-shot Class-incremental Learning. http://arxiv.org /abs/2206.10596 [7] Peng, Can et al. Few-Shot Class-Incremental Learning from an Open-Set Perspective, ECCV 2022 [8] Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip Torr, and Dacheng Tao. Neural collapse inspired feature-classifier alignment for few-shot class-incremental learning. In International Conference on Learning Representations 2023

Questions

- See weaknesses. - Discussions and comparisons to the most recent sota are necessary

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have adequately mentioned the limitations. And motivate the setting where the pretrain base classes being from different data distribution as the incremental new classes is actually a more realistic setting.