Summary
This paper proposes Continual Learning with Probabilistic Finetuning (CLAP) for class-incremental learning using CLIP. The key modules of the proposed idea are as follows. First, the authors introduce a CLIP-based probabilistic finetuning model using Bayesian Variational Inference to achieve better generalization during the continual learning process. Second, they propose a visual-guided attention (VGA) model to facilitate cross-modal feature alignment in the continual learning process. Lastly, to alleviate forgetting of pre-trained language-aware CLIP knowledge, they suggest past-task distribution generalization. Additionally, to enable parameter-efficient learning, a probabilistic adapter is used, along with a method for its initialization to enhance stability. Experimental results on various datasets demonstrate that the proposed algorithm achieves superior class-incremental learning performance compared to existing algorithms.
Strengths
The strengths of this paper are as follows:
1. The paper is well-written and easy to understand.
2. The proposed modules for successful class-incremental learning using CLIP are meticulously designed. Through various analyses and ablation studies, the roles of each module are thoroughly demonstrated. Although the algorithm appears somewhat complex compared to existing algorithms, parameter and time analyses show that the actual cost difference is negligible.
3. In class-incremental learning experiments using diverse datasets, the proposed algorithm consistently outperforms existing algorithms across various evaluations.
Weaknesses
1. I have no major concerns regarding the contribution of the proposed algorithm for class-incremental learning using CLIP. However, I have some questions based on the experimental results.
1-1) Unlike algorithms like L2P and DualPrompt, which utilize exemplar memory, the paper does not use exemplar memory. Are the results in L2P and DualPrompt sections of the paper based on using exemplar memory in a fair comparison? To ensure a fair comparison, results using exemplar memory should be shown.
1-2) The paper only considers prompt-based algorithms like L2P and DualPrompt using Vision Transformer (ViT). However, newer prompt-based algorithms (e.g., CODA-Prompt) that achieve better performance and representation-based algorithms (e.g., Ranpac and EASE) that generally improve performance have been proposed. The authors should consider these algorithms as additional baselines. For more details, please refer to this survey paper [1].
1-3) Among existing baselines, PROOF is currently the state-of-the-art algorithm. While Table 1 shows PROOF's results, Tables 2 and 16 do not. To validate the superiority of the proposed algorithm, results for PROOF should be shown in these tables and other key experiments.
2. Recent research has highlighted discussions on hyperparameter tuning for class-incremental learning algorithms [2] and pointed out issues regarding computational costs[3]. From this perspective, I believe that further discussion and additional analysis on computational costs regarding the proposed algorithm would make the paper more convincing.
3. Lastly, I have personal concerns regarding the necessity of class-incremental learning research using CLIP. As shown in [1], algorithms using only ViT have already achieved excellent performance in class-incremental learning research. Despite using pretrained ViT, one of their major advantages is achieving superior performance without using exemplar memory. Considering this perspective, the proposed research not only uses a pretrained CLIP model but also requires text information and additional exemplar memory usage. Simply comparing numerical experimental results suggests that the proposed algorithm does not achieve overwhelmingly superior performance compared to algorithms using only pretrained ViT. In light of this, I would like to understand: 1) What justifies the necessity of class-incremental learning research using CLIP with exemplar memory? Additionally, 2) I am curious about the potential for the proposed algorithm's ideas to be applied to continual learning in other domains using CLIP or continual pre-training of CLIP itself.
[1] "Continual learning with pre-trained models: A survey." arXiv preprint arXiv:2401.16386 (2024).
[2] "Hyperparameters in Continual Learning: a Reality Check." arXiv preprint arXiv:2403.09066 (2024).
[3] "Computationally budgeted continual learning: What does matter?." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
Questions
See the Weakness section. I don't have significant concerns about the algorithm proposed in this paper, but I couldn't give it a higher score due to some experimental uncertainties and questions about the necessity of the setting. If the authors address these concerns in the rebuttal, I would gladly raise my score.
Limitations
There is no potential negative societal impact of this paper.