Summary
This paper proposes a Singular Value Fine-tuning for few-shot Class-incremental Learning (SVFCL). SVFCL incorporates incremental adapters based on the pre-trained ViT encoder, which tries to reduce the risk of overfitting.
Weaknesses
I think this paper cannot capture the author's core idea, and there are many contradictions in this paper:
From the method:
1) In the abstract, the authors say that SVFCL contains only a small number of learnable parameters, but in Section 4, SVFCL fine-tunes the backbone of the foundation model. The authors use ViT-B/16-1K as the backbone, which has a large number of parameters.
2) I think the biggest challenge with fine-tuning the backbone is catastrophic forgetting, and fine-tuning the matrix S can retain the previous knowledge and adapt to the new task. However, the authors focus on overfitting and mention that the solution to overfitting is to reduce the number of parameters for fine-tuning in Section 4. Compared with SVFCL, other methods freeze the backbone and only fine-tune the classifier, which fine-tune fewer parameters and solve overfitting better.
3) I think the authors are vague about the description of the adapters. Where are adapters used? And whether there are additional parameters? In the contributions, adapters introduce quite a few learnable parameters, are they MLP? But in Section 4, the authors finetune the matrix S. If additional parameters are available, introducing the number of parameters will still exacerbate the overfitting.
4) The authors propose to fine-tune the backbone. Whether the entire backbone do the singular value fine-tuning, which undoubtedly needs to adjust a huge number of parameters?
5) The authors update the matrix S in the method. Why only update S? What is the effect if update U or V? what is the effect if update two matrices?
6) In the experiment, the authors mention the use of MLP. So will the MLP perform singular value decomposition? Will it fine-tune in each session? In addition, the authors add an MLP after 12 ViT blocks, which undoubtedly increases the number of parameters, will it further lead to overfitting?
From the writing:
1) The method focuses on FSCIL, but the authors spend a lot of space describing CIL in the abstract. In addition, the authors propose the use of adapters in the abstract and the contributions, but do not introduce this module in detail in Section 4.
2) Figure 1 does not indicate which dataset the example comes from, is it from the base session or the incremental sessions? And I think the author should include more examples in Appendix A instead of just a few more visualizations of other methods.
3) The expression of Figure 2 is very vague. The caption shows this figure represents the whole framework, but it only describes the weights for parts of the model. Meanwhile, the authors freeze the pre-trained parameters. What does the red part indicate? And where are all MLPs used by the author?
4) I think the first contribution is the same as the second contribution. The first contribution is only the application of the adapter, which appears limited and may not be entirely convincing.
5) The authors also have some inconsistent descriptions and grammatical errors, such as the authors describe SVFCL as a strategy in the contribution but a framework in Figure 2.
From the experiment:
1) The role of Figure 3 and the three tables is consistent, why do the authors repeat the experimental results?
2) I think it would be fairer for the authors to compare some FSCIL or CIL methods that use ViT or Transformer, such as [1-4].
[1] Pre-trained Vision and Language Transformers Are Few-Shot Incremental Learners. CVPR 2024.
[2] Calibrating Higher-Order Statistics for Few-Shot Class-Incremental Learning with Pre-trained Vision Transformers. CVPR 2024.
[3] Semantic-visual Guided Transformer for Few-shot Class-incremental Learning. ICME 2023.
[4] Ranpac: Random projections and pre-trained models for continual learning. NIPS 2023.
3) The authors use only 2 epochs in each incremental session, is the model underfitting? The authors use 3 epochs in the base session, whether the high accuracy relies on knowledge leakage from pre-trained models.
4) I think the authors should add the analysis of matrix U, S, and V in the Discussion, and what role U and V play in the model.