Task-recency bias strikes back: Adapting covariances in Exemplar-Free Class Incremental Learning

Summary

This paper addresses the Exemplar-Free Class Incremental Learning (EFCIL) challenge. It identifies two critical issues that undermine the effectiveness of existing methodologies and proposes a novel approach, AdaGauss, which adapts covariance matrices from task to task and mitigates task-recency bias.

Strengths

1. The paper is generally well-written and well motivated. It clearly demonstrates that the changes of covariance matrices also matters in Exemplar-Free Class Incremental Learning. 2. The proposed method demonstrates pretty good experimental results on all five datasets whether training from scratch or starting from a pre-trained backbone. 3. The proposed method is straightforward and easy to understand.

Weaknesses

1. I appreciate the efforts the authors have devoted to detailing the three observations encountered in Exemplar-Free Class Incremental Learning. However, observations 2 and 3 appear quite similar to me, as both seem to represent a simplification of the representation for previously seen classes. Additionally, the analysis of observations 2 and 3 does not significantly depart from existing explanations for why classification results tend to skew towards recent tasks, a phenomenon (task-recency bias / representation forgetting) already well-documented in the broader field of continual learning. 2. The proposed idea that both the mean and covariance should be adapted during training shares similarities with test-time adaptation methods. Therefore, some comparisons are necessary to delineate these relationships further. 3. Although it is quite straightforward that encouraging the feature extractor to produce features with linearly independent dimensions can mitigate dimensionality collapse, this approach does not guarantee the production of meaningful features. Additionally, it remains unclear whether simply optimizing the covariance matrices of features from one mini-batch can ensure linear independence. Some theoretical analysis would better explain the effectiveness of the proposed loss term $L_{AC}$ in mitigating dimensionality collapse (which I think there do exist some possible results to derive).

Questions

See the weakness.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

N/A

Summary

Existing methods use Gaussian distributions to represent classes in the feature extractor's latent space, but face unchanged covariance matrices and task-recency bias. This paper introduces AdaGauss, an approach that adapts covariance matrices and mitigates the bias through an anti-collapse loss function. AdaGauss achieves top performance on EFCIL benchmarks and datasets, whether training from scratch or using a pre-trained backbone.

Strengths

1. The paper presents innovative approaches to tackle the EFCIL problem. 2. The writing is quite good, and easy to follow overall. 3. he paper includes comprehensive experimental findings to support its claims.

Weaknesses

1. Literature is incomplete. The paper concentrates on the EFCIL, yet quite a number of important EFCIL techniques [1-3] are not presented, and I want to see experimental comparisons with [1,3] if possible. [1]Huiping Zhuang, Zhenyu Weng, Hongxin Wei, Renchunzi Xie, Kar-Ann Toh, Zhiping Lin”ACIL: Analytic Class-Incremental Learning with Absolute Memorization and Privacy Protection”, Thirty-Sixth Conference on Neural Information Processing Systems (NeurIPS) 2022. [2] Ma, C.; Ji, Z.; Huang, Z.; Shen, Y.; Gao,M.; and Xu, J. 2023. Progressive Voronoi Diagram Subdivision Enables Accurate Data-free Class-Incremental Learning. In The Eleventh International Conference on Learning Representations. [3] Zhuang H, He R, Tong K, et al. DS-AL: A dual-stream analytic learning for exemplar-free class-incremental learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(15): 17237-17244. 2. The claim "dimensionality collapse" concept is very important, but the authors didn't explain anything related. 3. An adaptor is needed for implementing the proposed algorithm. Could you provide a comparison among the proposed methods regarding the auxillary networks imposed?

Questions

see weakness.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

n.a

Summary

The paper addresses the problem of Exemplar-Free Class Incremental Learning (EFCIL), which involves training a model on sequential tasks without access to past data. Current methods represent classes as Gaussian distributions in the feature space, enabling Bayes classification or pseudo-feature replay for classifier training. However, these methods face issues such as the need to adapt covariance matrices after each task and susceptibility to task-recency bias due to dimensionality collapse. The authors propose AdaGauss, a novel method that adapts covariance matrices between tasks and incorporates an anti-collapse loss function to mitigate task-recency bias. AdaGauss achieves state-of-the-art performance on popular EFCIL benchmarks and datasets, whether training from scratch or using a pre-trained backbone.

Strengths

Overall this is a good paper with the following strengths: * This paper provides a clear experimental analysis and elaboration of the motivation for the proposed method. * For EFCIL, it is important and interesting to track changes in the distribution of past classes. * The proposed method achieves good results in multiple settings.

Weaknesses

* The method of knowledge distillation through projectors proposed in this paper has been widely studied and discussed [1, 2]. Different from the general field of knowledge distillation, what are the technical innovations of the proposed method? Does it have unique insights for continual learning tasks? * The proposed method introduces several additional structures and requires a large number of samples from the simulated distribution to be trained, so the number of model parameters and the required training time should be discussed. [1] Chen Y, Wang S, Liu J, et al. Improved feature distillation via projector ensemble. NeurIPS 2022. [2] Miles R, Mikolajczyk K. Understanding the role of the projector in knowledge distillation. AAAI 2024.

Questions

I have some questions mainly about the experiments: 1. The experiments in this paper use the "learn from scratch" setting. Another common setting used in CIL is the "learn from half", which is also a dominant setting in EFCIL. It simply means that half of the classes in the dataset are learned as an initial task, and then the remaining classes are evenly divided into subsequent incremental learning tasks. This experimental setting is used in most of the compared works, such as PASS, SSRE, FeTrIL and FeCAM. How does AdaGauss perform in this setting? 2. Although the authors provide a detailed ablation analysis in Table 3, I still have some concerns. Why is $L_{AC}$ replaced by "utilized covariance matrix shrinking with the value of 0.5" in the sixth row of table 3, instead of just removing it? Are there other components of the proposed method that cannot be run in the absence of $L_{AC}$? 3. Does the adaptive tuning for Gaussian distributions fail if there is no $L_{AC}$ to resist dimensionality collapse? 4. Additionally, it would be beneficial if the authors could include a row comparing the performance of AdaGauss without both "Adapt mean", "Adapt covariance" and $L_{AC}$. This additional row would contribute to a clearer comparison and enhance the transparency of the evaluation.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

It is hard to foresee any potential negative societal impact of this theoretical work.

Summary

This paper analyzes the impact of dimensionality collapse in EFCIL and examines the distribution shift of the mean and covariance matrix. Based on these findings, the paper proposes the AdaGauss method, which adapts the covariance and mean, and designs a loss term to prevent the dimensionality collapse of the feature extractor.

Strengths

- The analysis of the dimensionality collapse of the feature extractor is interesting. - The paper provides a comprehensive experimental analysis, which clearly demonstrates the effectiveness of different components within the proposed model.

Weaknesses

- The writing in the current version needs improvement. There are many parts that I had to reread several times to understand. For example, the three observations in Lines 89-118, I strongly recommend to summarize them in one sentence initially to highlight the core findings. In Section 3.3.2, which is one of the most important parts and contributions of the paper, the explanation is insufficient even though I refer to the appendix. What is the meaning of $a_i$ ? There is no clear definition. - Regarding Figure 1, is it merely an illustrative figure, or is it a visualization figure? If it is a visualization figure, what are the details used to depict this figure? - In Section 3.3.4, the learned adaptation network is trained on the t-th task. Why can it be used for previous μ and σ? The underlying assumption is that all previous tasks share the same shift pattern. Is this a reasonable assumption? - For Algorithm 1, I am not sure how it addresses the Batch Norm statistics, which are crucial to continual learning and the covariance shift this paper focuses on [1,2]. - The time complexity of the proposed method compared to other methods should be discussed. [1] Continual Normalization: Rethinking Batch Normalization for Online Continual Learning, ICLR 2022. [2] Diagnosing batch normalization in class incremental learning, 2022.

Questions

1. Please address the questions in the weakness. 2. Can the proposed method be applied to other network structures such as 4-layer convolutional networks or ViT?

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

NA

Thank you for the response

Thank you for the response. The rebuttal has addressed my concerns and I will keep my rating.

Response

I appreciate the authors taking the time to respond and I tend to keep my score.

Comment by Reviewer xJqZ

I thank the authors for the efforts to address the concerns, especially the additional experiments conducted to verify the effectiveness of the proposed method with different backbone networks. The clarifications made by the authors and the results of the additional experiments addressed my concerns. I have increased my score to 6 accordingly.