FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learning

Summary

This paper addresses the problem of continual learning, which holds great importance in the machine learning field. The authors argue that using the Mahalanobis distance is more optimal than the Euclidean metric for handling new classes. The proposed method is evaluated on various continual learning settings and compared against state-of-the-art approaches.

Strengths

1. Continual learning is of great importance in the machine learning field, and the availability of code is appreciated. 2. The proposed method is evaluated on multiple continual learning settings and compared against state-of-the-art approaches. 3. The figures effectively illustrate the distribution of old and new classes, making the proposed approach reasonable.

Weaknesses

1. The largest concern about this paper is the assumption of pre-trained model (or say, relying on large number of classes in the first task). There are two typical settings of class-incremental learning, training from scratch and training from half. The former equally assigns classes into each incremental task, while the latter treats half of the total classes in the first task. According to a recent work [1], these different settings concentrate on different aspects of the continual learning algorithms. In this paper, the experiments (as well as empirical observations) are conducted on the training from half setting, making it less convincing on the other setting. Would the proposed method work for the other setting? Will the observations of covariance be the same on the former setting? More empirical evaluations and experimental comparisons are essential. 2. In figure 1, why does the training process lead to the gaussian-like distribution of the seen classes and non-gaussian-like distribution of unseen classes? Perhaps some discussions about theoretical insights behind such phenomena should be addressed. 3. The current proposed method is a combination of several mature tricks, e.g., Mahalanobis distance, Covariance Shrinkage, and Tukey’s transformation. I understand that the most contributions lie in the motivations in the preliminary part, while the combination still makes the novelty/contribution to share with former works. [1] Online Hyperparameter Optimization for Class-Incremental Learning. AAAI 2023

Questions

Remarks 1. In Figure 5, clarification is needed regarding how the authors managed to sample 20,000 instances per class for CIFAR100, as the maximum number of instances per class is 500 for CIFAR. 2. What is “classifier incremental learning” in Line 137? 3. It would be beneficial to provide an implementation guideline with pseudo code. In summary, this paper addresses an interesting phenomenon in the field of continual learning and proposes a simple baseline method to address it. The proposed approach is evaluated on various protocols, including many-shot CIL, few-shot CIL, and pre-trained CIL, demonstrating superior performance compared to state-of-the-art methods. Although there are some evident drawbacks, the novel findings presented in this paper contribute to the continual learning field. Therefore, my initial rating is a borderline accept.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

NA

Summary

The authors claim that feature space of a learnt network with old classes is more scattered in newly learned classes and this prevents Euclidean distance to represent the distribution properly. Therefore, they have adopted Mahalanobis distance instead to handle the heterogenety of the newly coming class distribution. To boost up the performance, covariance normalization, covariance shrinkage, covariance matrix approximation have been adopted.

Strengths

- The performance presented in this paper is state of the art. Even though bayesian classifier is usually only valid for classification problem, it is still impressive to see the advance in performance. - The settings and amount of experiments in this paper are rich enough to support the hypothesis claimed in this paper. - Especially, thorough ablation over the components of this method is very interesting.

Weaknesses

- many existing techniques comprising the proposed method make this paper somewhat short of novelty but I still think the whole idea throughout this paper is reasonable. - In Eq (6), I dont understand why the norm of Y is used all over the equation. Y should be the target of all samples and the norm of this does not seem to be important. I think more explanation about how this equation is derived is required here.

Questions

- I agree that a bayesian classifier is well suited with incremental learning. However, when it comes to applying this incremental learning procedure to other kinds of tasks such as segmentation or generative models, will it be still effective to use it? This is not a critical comment and I just want to know the opinion of the authors.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

No comments required here.

Summary

This manuscript introduces a Bayes classifier method FeCAM to address the feature distribution shift within the realm of Continual Incremental Learning (CIL). This method uses Mahalanobis distance formulation and additionally uses some techniques like correlation normalization, covariance shrinkage, and Tukey’s transformation to estimate better covariance matrices for continual classifier learning, thereby enhancing the efficacy of the model in addressing CIL tasks.

Strengths

The manuscript presents clear and logical theoretical reasoning, making it easily understandable for readers. Moreover, the method is firmly grounded in a well-defined theoretical framework.

Weaknesses

1. There exist some mistakes, for example, on line 224, it states “20 initial classes and 10 IL steps of 5 classes”. It should be corrected to “20 initial classes and 10 IL steps of 8 classes”. 2. From table 1, it can be observed that the results of the FeTrIL are recomputed, whereas the other results are reproduced using the original configurations of the methods. This deviation contradicts the manuscript 's description. In addition, please explain how the experimental setup in this manuscript differs from that of FeTrIL. 3. On Page 7, the datasets “Split-ImageNet-R”, “Split-CIFAR100”, “CoRe50” and the “domain-incremental learning” need to be further explained. In particular, it is important to explain the differences between Split-CIFAR100 and the previous CIFAR100 dataset. 4. From ablation Studies, it can be observed that the experimental results in row 7 of Table 4 outperform those in row 6. Therefore, when the diagonal matrix is combined with "Tukey", "Shrinkage" and "Norm", whether the performance will be better.

Questions

1. Page 1, ‘Recent approaches to incrementally learning the classifier by freezing the feature extractor after the first task have gained much attention.’ I think it’s necessary to explain the weakness of the method after this sentence. 2. It can be observed that the proposed method FeCAM outperforms the existing state-of-the-art (SOTA) results on all datasets under the few-shot settings. But there should further analysis to explain the advantage of the proposed FeCAM.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The proposed method relies on a pretrained network to learn good representations because the method does not learn new features but reuse the ones learned on the first task. Therefore, when training from scratch, starting with small tasks, this method may not work well.

Summary

The authors study classifiers in the incremental learning scenario with a fixed strong pre-trained feature extractor. They point out the limitations of the Euclidean distance-based nearest class mean classifier in continual learning and demonstrate the benefits of Mahalanobis distance-based classifier. Several methods (shrinkage, Tukey’s transformation, covariance normalization) are proposed to estimate the covariance matrices and their effectiveness is examined through ablation studies. The authors evaluate their method in various settings, including many-shot CIL, few-shot CIL, and domain incremental learning. Although the proposed method does not require saving previous samples, it outperforms the existing CL methods including those based on replay, in terms of performance.

Strengths

1. The proposed method is simply, yet outperforms the existing methods 2. The proposed method does not require saving previous samples 3. The paper is well-written and easy to follow

Weaknesses

1. My main concern regarding this method is that it relies on a fixed feature extractor after the initial training, with only the classifier being adapted. As [1] shows, fixed models underperform when the pre-training data and the continual learning data are dissimilar because they cannot acquire new knowledge. It’d make the paper stronger if the authors show how the model performs with a feature extractor pre-trained with dissimilar classes from CL. For instance, [2] uses a model pre-trained with ImageNet after removing the classes similar to CIFAR and Tiny-ImageNet. 2. Saving the covariance matrix for each class in the feature-level seems to be expensive for memory, especially when there are a large number of classes. [1] Ostapenko et al., Continual learning with foundation models: An empirical study of latent replay. CoLLAs, 2022 [2] Kim et al. A multi-head model for continual learning via out-of-distribution replay. CoLLAs, 2022

Questions

1. How does the performance change when using models pre-trained using classes dissimilar to the classes used in CL? 2. Can the authors provide any suggestions on how to apply their method for a trainable feature extractor?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Refer to Weaknesses and Questions.

Summary

The paper presents a method for class-incremental learning when a strong base classifier is available and incrementally classes are added to the classifier. The base classifier is kept frozen and another classifier is used on top of the features from this classifier. Instead of using deep networks, authors propose using a bayes classifier and modeling the distance between classes using mahalanobis distance instead of the popular euclidean distance. Authors argue and show that the features of the incoming new classes are heterogenous and are better suited to be measured with mahalanobis distance. The authors also propose some solutions to the problems with the covariance matrix in mahalanobis distance calculation and to improve the performance further. The method is evaluated on many and few-shot settings and evaluated on many standard datasets where the method seems to beat many existing methods.

Strengths

1. The paper is very well-written and easy to follow, supplementary material also provides good background and completes the picture, 2. Authors do a good job in motivating and validating their motivation and methodology, with plots, figures and qualitative numbers. The use of mahalanobis distance and corresponding proposals to the covariance matrix seem like simple but powerful changes. 3. The method works without additional training of classifiers so it would be quicker to incorporate new classes in an incremental setting 4. It does not use significant additional storage space and the quantitatively the methods does well compared to existing methods

Weaknesses

1. There are no error bars added for the results which would provide more confidence in the results, it is a convention in incremental learning research to only have a single plot per method without any error bars, earlier since it used to be computationally expensive but no reason we should start doing it now, 2. The limitations already address this, but a follow-up on changing features would exacerbate the drift problem and would love to see the theory handle that case as well, the authors already list as future work so no action is needed for this submission,

Questions

1. How is the order of the classes which are added decided? 2. Which classes are selected as part of MiniImagenet and ImagenetSubset, the classes and few-shot images should also be provided for full reproducibility

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Authors have sufficiently addressed the limitations of the method. The limitations make the method only applicable in certain cases but the authors acknowledge it and the performance of the method seems pretty good in that setting.

Response to the authors

I thank the authors for the detailed response, which addresses most of my concerns. Although I find the updated figures unavailable, the response contains most of the results. I update my rating to weak accept. BTW, it is still very important to report the full results given various settings (e.g., training from half and scratch), and I expect the authors to report more results on the second setting in the final version. It at least answers the capability and incapability of the proposed method in various CIL settings.

Response to the author comments

Thank you for the detailed responses. My concerns are addressed. I raised my score from 5 to 6. Please reflect the discussions and experiments in the revision.