Summary
The paper addresses the problem of Class-Incremental Learning by introducing a Hybrid Autoencoder (HAE). The proposed model is designed for both exemplar replay during incremental training and classification at inference. The autoencoder consists of two components: an encoder, which is trained to minimize the Euclidean distance between the latent representation and the corresponding class centroid, and a decoder, which is trained to minimize the reconstruction error between the input and output images. Both the encoder and decoder are trained with a distillation loss on the previous task to mitigate activation drift.
At the encoder level, the class centroids serve as anchors in the latent space, guiding the latent representations towards their respective classes. These centroids are initialized before training using the Charged Particle System Energy Minimization (CPSEM) method, which ensures that the centroids are well-separated. After training, a nearest-mean classification rule is applied to classify test images based on the proximity of their latent representations to these centroids.
In the post-training phase, the encoder's latent space output is used to populate a replay buffer with latent representations of the current task samples, following a herding strategy. These representations are replayed in the next task by feeding them into the decoder trained on the previous task, which generates the corresponding images. These generated images are then used for training on the new task. Instead of storing images, as is typically done in incremental learning, the proposed method reduces memory requirements by storing only the latent space representations.
The authors compare their approach to several incremental learning methods on benchmarks such as MNIST, Balanced SVHN, CIFAR-10, and miniImageNet. They also analyze the method's resource consumption, evaluate different decoder sizes, and provide an ablation study by comparing its performance when real images are used during the replay phase instead of the generated ones.
Weaknesses
Overall, I believe the presentation of the paper requires significant improvement. Below are my major concerns regarding the presentation:
- (a) The complexity analysis in the introduction needs to be clarified and expanded. The notation $O(0.1t)$ is incorrect according to the definition of Big-O notation. What does this represent? If the authors intend to convey that memory is saved by storing the latent space representation, I suggest incorporating both the latent space dimension and image size into the complexity analysis. Additionally, the term $e$ in $O(cte)$ is not defined and requires clarification.
- (b) The notation throughout the paper is difficult to follow, with multiple indices used unnecessarily (e.g., in Equation 1). This excessive use of notation makes the paper hard to read. I suggest simplifying the notation wherever possible.
- (c) Equation 1 seems to imply that all examples from previous tasks are needed to minimize the reconstruction error. However, I understand that this is not the case—some examples are real images, while others are generated. The replay buffer should be explicitly highlighted to clarify this in the equation.
- (d) The three pseudocode blocks on page 4 make the methodology difficult to follow. Including all three pseudocode blocks on a single page compresses the accompanying methodology description into less than half a page. As a result, some LaTeX formulas in the main text break across two lines, further increasing the difficulty of reading.
- (e) The organization of the paper should be reconsidered. Given the limited space for submission, dedicating more than two pages to the literature review while allocating just over one page to the methodology does not allow for a proper description of the proposed approach. I suggest moving the extended literature review to the appendix and presenting a more concise version in the main paper.
Regarding the methodology and experimental section, my major concerns are as follows:
- (f) The introduction of the Charged Particle System Energy Minimization (CPSEM) for initializing class centroids is interesting but requires additional explanation. It is unclear why this type of initialization benefits the autoencoder and how it relates to the Coulomb interaction energy . While I do not expect a full background on the calculus of variations for minimizing energy, more mathematical details—even in the appendix—would be helpful. An analysis of how centroids are distributed in the latent space is required to underline why the proposed strategy is effective. Furthermore, the CCE (class centroid embedding) placement is explained only through pseudocode, with no accompanying description. At a minimum, the operations performed in Algorithm 2 should be explained in words to provide intuition, especially for readers unfamiliar with the physics-based intuition behind this algorithm.
- (g) Regarding CCE, why is this initialization considered effective? If the goal is to initialize centroids such that the class centroids are distant from each other, why not simply use the K-means algorithm? Alternatively, why not select class centroids as the latent vectors for each class that are the most distant from each other in terms of Euclidean distance, in a similar way as performed with hard negative sampling?
- (h) The usage of the latent space for memory reduction and the decoder for latent replay is not novel. For example, Ayub et al. (ICLR 2021) [1] employ the encoder for storing latent representations and replay these latent representations in subsequent incremental learning steps. When the memory budget is reached, the latent representations are compressed into centroids and covariances. A comparison with their approach is necessary. The storage of latent representations for incremental learning and efficient memory replay with autoencoder is also explored in [2].
- (i) Comparison in Table 2. 1) Some comparisons are unnecessary since the methods the authors compare to perform different tasks. For instance, Prediction Error Based Classification (PEC) [3] is designed for online continual learning (single-pass data), while the authors address the problem of offline incremental learning. It is clear that the harder setting for which PEC is designed results in lower performance compared to the authors' method. The authors should compare their approach only with offline class incremental learning methods under the same conditions. 2) Since the authors' method operates in an offline incremental learning setting, they should compare it with recent exemplar-based class incremental approaches, such as X-DER [4] and MEMO [5]. Additionally, the comparison should consider using a larger and more realistic backbone, such as ResNet-18, which is now commonly evaluated with more parameters and a larger latent space [4][5]. The paper should also evaluate how the method performs with higher-resolution images (e.g., 224x224) on a dataset like ImageNet100 [5].
- (l) In Table 3, the authors report the wall-clock time for training their methods. They state that training takes about 8 hours on CIFAR-100 with ResNet-32, which seems excessive. In FACIL [6], joint training on a not particularly novel GPU requires less time, as only about 400k parameters need to be optimized. What are the timings for joint training with the same epoch budget? An incremental learning method should be more efficient than joint training. Additionally, the authors should specify the device used for the experiments when reporting training time.
[1] A. Ayub and A. Wagner, “{EEC}: Learning to encode and regenerate images for continual learning,” in International
Conference on Learning Representations, 2021.
[2] Caccia, L., Belilovsky, E., Caccia, M. & Pineau, J.. (2020). Online Learned Continual Compression with Adaptive Quantization Modules. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research.
[3] Michał Zaj ˛ac, Tinne Tuytelaars, and Gido M van de Ven. Prediction error-based classification for
class-incremental learning, in ICLR 2024
[4]Zhou, Da-Wei and Wang, Qi-Wei and Ye, Han-Jia and Zhan, De-Chuan, A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning, in ICLR 2023
[5] Matteo Boschini, Lorenzo Bonicelli, Pietro Buzzega, Angelo Porrello, Simone Calderara. Class-Incremental Continual Learning into the eXtended DER-verse, in TPAMI 2022
[6] Marc Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D. Bagdanov, Joost van de Weijer Class-incremental learning: survey and performance evaluation, TPAMI 2022
Questions
Overall, I believe the paper has significant issues with presentation (as outlined in points [a-e] of the weakness section), which necessitate a major reformatting. Additionally, regarding other concerns, the novelty appears limited to performing classification at the encoder level and the introduction of CPSEM for initializing class centroids. While the latter is a novel contribution, it is neither well-explained nor well-motivated (as noted in points [f-g] of the weakness section). The experimental section, particularly the method comparison, needs refinement by considering recent related work on the proposed approach, utilizing higher-resolution benchmarks, and employing architectures with a larger latent space (as indicated in points [h-i] of the weakness section). Moreover, the details about the training resources should be clarified (point [l] in the weakness section).
I believe the paper has the potential for significant improvement for a future submission. To enhance its novelty, the authors should focus more on the description and proposal of the class centroid initialization, which could provide both theoretical and empirical insights. However, as previously mentioned, these insights are lacking in the current version. Additionally, improving the experimental section would further strengthen the submission.
Considering all the above, I recommend rejecting the current submission. The paper is not yet ready for publication and requires significant revisions. I suggest making these improvements and submitting it to a different venue.