Pre-trained deep neural networks (DNNs) are being widely deployed by industry for making business decisions and to serve users; however, a major problem is model decay, where the DNN's predictions become more erroneous over time, resulting in revenue loss or unhappy users. To mitigate model decay, DNNs are retrained from scratch using old and new data. This is computationally expensive, so retraining happens only once performance significantly decreases. Here, we study how continual learning (CL) could potentially overcome model decay in large pre-trained DNNs and greatly reduce computational costs for keeping DNNs up-to-date. We identify the "stability gap" as a major obstacle in our setting. The stability gap refers to a phenomenon where learning new data causes large drops in performance for past tasks before CL mitigation methods eventually compensate for this drop. We test two hypotheses to investigate the factors influencing the stability gap and identify a method that vastly reduces this gap. In large-scale experiments for both easy and hard CL distributions (e.g., class incremental learning), we demonstrate that our method reduces the stability gap and greatly increases computational efficiency. Our work aligns CL with the goals of the production setting, where CL is needed for many applications.
Paper
References (100)
Scroll for more · 38 remaining
Similar papers
Peer review
Summary
The paper studies the problem of stability gap [1] in the context of efficient continual learning with deep neural networks. As an effort to understand and mitigate the phenomenon, the work validates two hypotheses and their equivalent solutions: (a) the stability gap arising from larger loss values due to new output classes -- the mitigation strategy of which involves initializing the new class output class units with the mean of unit length class embeddings and using soft-targets for training the network; (b) the gap arising from excessive network plasticity in which case the level of plasticity of the different network layers are controlled in a dynamic fashion. The algorithmic contribution is demonstrated with a range of experiments using a pre-trained ImageNet-1K model. References: [1] De Lange, Matthias et al. “Continual evaluation for lifelong learning: Identifying the stability gap.” ArXiv abs/2205.13452 (2022): n. pag.
Strengths
1. The work proposes a number of strategies to alleviate the negative effect of the stability gap. As such, their target domain involves the parameter regularization space (weight init, limiting the plasticity) as well as the function regularization space (using soft targets). 2. The simplicity of the proposed weight initialization strategy is indeed impressive in light of its contribution towards bridging not only the stability gap but also the plasticity gap. 3. Detailed supporting experiments considering the diverse settings of non-rehearsal, rehearsal, memory constraints, backbone architectures, etc.
Weaknesses
1. As different weight initialization techniques tend to be prone to their specific limitations (e.g., a saturation of the activation function, suitability to sigmoid vs softmax activation, etc.), it would be interesting to see how the proposed initialization method stands against other existing ones (Kim's [2] for CL, He's [4] for non-CL) under these circumstances. Such a study could be beneficial to the continual learning community in understanding when (not) to use the proposed init method. 2. The main table of results lacks a proper comparison with the existing literature [3]. 2. Since LoRA restricts the number of trainable parameters in hidden layers, I believe that it might be susceptible to situations where if the lower layer parameters have been frozen at the earlier tasks, then it might affect the network's capacity to learn new low-level feature compositions for later tasks. This is indeed one of the reasons why freezing lower-layer features is not preferable in continual learning [2]. Can the authors comment on this? 3. What concerns me is the efficacy of the proposed SGM method in terms of the computational overhead involved. While the work compares the computational efficiency in terms of the number of iterations required to overcome the stability gap (Figures 1 and 3), there seems to be no mention of the more common measures of computational overhead (e.g. complexity, training time). For a practitioner, a few additional iterations of training might be preferable to a complex training algorithm with a lesser number of iterations yet a significantly longer training time. Can the authors comment on this? 4. In Figure 3, it would have been better to also show the performance graphs in comparison to the rehearsal-based methods of Table 3 (DERpp and GDumb). References: [2] Kim, Sohee and Seungkyu Lee. “Continual Learning with Neuron Activation Importance.” International Conference on Image Analysis and Processing (2021). [3] Li, Depeng et al. “IF2Net: Innately Forgetting-Free Networks for Continual Learning.” ArXiv abs/2306.10480 (2023): n. pag. [4] He, Kaiming et al. “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification.” 2015 IEEE International Conference on Computer Vision (ICCV) (2015): 1026-1034.
Questions
Please see the weaknesses section. Overall, this is an interesting work in the direction of training efficiency in CL. However, my major concerns remain with the limited experimental comparisons with the existing literature. Given these clarifications in the rebuttals, I would be willing to increase the score.
Rating
5: marginally below the acceptance threshold
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Thank you for taking the time to review our paper. # Compare Against More Weight Initialization Methods Based on our hypothesis, data-driven weight initialization methods that reduce the loss would make the most sense. That said, we did compare against Kaiming (He) initialization, which the reviewer suggests, since it was used by vanilla rehearsal for the output layer throughout experiments. We added this to supplemental materials. # The Main Table Should List SoTA Results We strongly disagree. We are aiming to do science and test hypotheses, and Table 3 reports the needed information to do this. For testing our hypotheses, adding many methods that do not use the same pre-trained backbone or keep other variables constant would be scientifically unjustifiable. Moreover, we are not aiming to compete with the SoTA, but to demonstrate the broad applicability of SGM for mitigating the stability gap. We did a fair comparison when we combined SGM with other existing CL algorithms e.g., DERpp, GDumb, REMIND, LwF which are widely used CL baselines. All compared methods used the same pre-trained model. We wanted to cover a variety of CL algorithms: 1) rehearsal based: GDumb, 2) rehearsal and knowledge distillation: DERpp, 3) non-rehearsal and knowledge distillation: LwF, 4) online CL: REMIND. Of course there are many more different CL methods but experimenting with all CL methods is intractable. We want to demonstrate that our method mitigates the stability gap and increases computational efficiency in most common CL methods. # Concerns Regarding Restricted Plasticity Prior work [1-3] found that earlier layer representations are universal and transfer well across tasks. Following prior work, we keep earlier layers frozen. However SGM does not rely on freezing earlier layers. SGM can be applied the same way even if we do not freeze earlier layers and want to update those layers. Regarding the paper that the reviewer cited that points to freezing lower-layer features may not be preferable, we note that the result of the paper they mention may not be broadly applicable outside of the small scale experiments conducted in that paper on CIFAR and MNIST, which is why their conclusions differ from those of others [1-3]. When searching for the paper they mentioned, we note that the ICLR reviews for the paper noted numerous concerns with the paper that do not look to have been addressed in the version accepted to ICIAP. # Concerns Regarding Computational Efficiency We agree that including timing experiments would be helpful beyond network updates. We will provide these numbers in a revised version of the paper in terms of FLOPS and hours. # Include Variants of Figure 3 for DERpp and GDumb We shall do this in a revised version of the paper. # References 1. Ramasesh et. al., Anatomy of catastrophic forgetting: Hidden representations and task semantics. In ICLR 2021 2. Zhao et. al., What makes instance discrimination good for transfer learning? In ICLR 2021 3. Sarfi et. al., Simulated Annealing in Early Layers Leads to Better Generalization, In CVPR 2023
Thank you for your response. This answers my question regarding restricted plasticity and weight initialization. As part of this review, I tried reimplementing the data-driven weight initialization technique for CL with pre-trained models and it indeed remains effective at stabilizing the initial loss values besides improving the performance slightly over vanilla. However, I am still concerned about the choice of methods for the comparison of the results. What I understood from one of the responses is that the authors chose the ConvNext backbone because it remains comparatively new and computationally efficient. However, in response to my concerns, the authors mention the unjustifiability of "adding many methods that do not use the same pre-trained backbone". Is this not a self-imposed limitation then? In particular, using a more common backbone (ResNet/ViT) for detailed comparison and using more recent architectures to advocate up-to-date CL applications can go hand in hand. Also, providing an updated Fig. 3 would be crucial for the comparison as an important goal for the work seems to be to enable more computationally efficient training. Further on that note, reporting more common measures of computational efficiency would help better justify the efficacy of the method. P.S. Existing literature **does not** imply "SoTA" and my review nowhere mentions the word "SoTA". Thank you for sharing the blog post on real-time continual learning.
To clarify the SoTA comment, we are aiming to control variables such that we are only comparing with and without a mitigation intervention, keeping all other variables constant. Many papers across deep learning, including many in continual learning, fail to keep things fair in their analysis. Table 1 is intended to be rehearsal alone vs. various interventions to test our hypotheses, where all other variables are kept constant, such that we are doing sound science. Also, we note that the IF2Net paper was only submitted to arXiv in June 18, 2023, and we have not found a peer reviewed version. LoRA is not compatible with ResNet architectures. In general, it can be used with fully connected layers, which are equivalent to the 1x1 convolutions. This means LoRA can be used with ConvNext and ConvNextV2, because it has many 1x1 convolutional layers. ResNet only has a fully connected layer as its output layer. We could potentially do an additional analysis with a ViT model if that would alleviate the reviewer's concerns; however, time is quite limited at this point in time to complete it during the rebuttal period. We can try is that would be helpful to the reviewer. However, we would like clarification regarding precisely what is being asked. We do compare in Table 8 against a Vision Transformer, where we found that SGM worked better than vanilla rehearsal. Is the reviewer asking for more experiments for more ViT models?
Timing Analysis
Timing analysis is somewhat nuanced. SGM requires very little overhead vs. vanilla; however, the analysis is complicated by the fact that vanilla rehearsal never fully recovers performance on ImageNet regardless of the number of updates and vanilla rehearsal has much lower results on new classes than SGM as well. We are trying to think of the best way to do this analysis. However, we can say that updating the model in our main results with SGM requires only 40 minutes whereas conventional fine-tuning on the joint ImageNet-1K + Places dataset requires 12.1 hours on the same hardware. We note that a critical feature of Figure 3 is that SGM's curve has a downward slope, except at the beginning, which is much steeper than vanilla rehearsal, indicating that SGM is becoming a more efficient learner over time.
Summary
In this paper, the authors aim to analyze and address the stability gap in class incremental learning. They put forth two hypotheses: first, that the stability gap arises partly from a significant loss at the output layer for the new classes, and second, that it is exacerbated by excessive network plasticity. The authors introduce two new metrics and conduct a series of experiments to investigate these hypotheses. Building on these ideas, they propose a novel method SGM to mitigate the stability gap. SGM combines four distinct techniques: (1) initializing weights of the output layer for new classes, (2) employing dynamic soft targets when learning a new task, (3) utilizing a network adaptor to limit the number of learnable parameters in hidden layers (LoRA), and (4) freezing weights in output layer for previously encountered classes. Techniques (1) and (2) target a reduction in loss for new classes, aligning with the first hypothesis. On the other hand, (3) and (4) aim to curtail excessive plasticity, in line with the second hypothesis. The authors present an extensive set of experiments to validate the efficacy of their proposed method.
Strengths
- The exploration of the stability gap is a compelling avenue that merits further exploration. - The authors have undertaken thorough experiments to substantiate their contributions.
Weaknesses
- In section 3.1 (second paragraph), the rationale behind setting f=0.3 is not clearly elucidated. A more detailed explanation in the experimental section would be beneficial. - While the main experiments utilize ConvNeXtV2-Femto, which has undergone unsupervised pertaining on ImageNet1k followed by supervised fine-tuning, it would be advantageous to also present results for training ResNet18 from scratch. This is important given the common use of ResNet18 in the CL community. - Table 1 could benefit from including results from normal fine-tuning as a lower-bound reference. Additionally, explicitly reporting accuracy on ImageNet-1k is recommended due to the class imbalance between ImageNet-1k and Places365-LT. In light of this, could the authors discuss whether stability takes precedence over plasticity due to this imbalance? Furthermore, given the context of continual learning with a pre-trained model on a large dataset like ImageNet-1k, it would be beneficial to discuss why other relevant methods [1-3] were not investigated. - Related to the dataset imbalance issue, it would be interesting to examine and analyze the stability gap for the initial batch of Places365-LT. [1] Learning to Prompt for Continual Learning, CVPR 22 [2] DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning, ECCV 22 [3] A Closer Look at Rehearsal-Free Continual Learning, CVPR 23 workshop
Questions
While the exploration of the stability gap is intriguing, my primary concern lies in the generalizability of the conclusions drawn from the experimental results, given the use of imbalanced datasets and an unsupervised pre-trained model with a large dataset. Even with the inclusion of experiments using a supervised pre-trained model in section 6.3, the findings may still lack generalizability, as the model might inherently learn more general features with a sufficiently large dataset. For example, this type of pre-trained model may possess good plasticity (transferability) without extensive weight updates. I would recommend the authors conduct experiments with a random initialized model and balanced tasks for a more comprehensive evaluation. Looking forward to the authors' response.
Rating
3: reject, not good enough
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Thank you for taking the time to review our paper. # Compare Against Other Methods that Use ImageNet-1K Pretrained Models We agree that it would be interesting to study some of these methods such as DualPrompt. We intend to do this, but it is not clear if we would have sufficient time during the rebuttal period to get it running. # What's the Rationale for $f = 0.3$? Lower $f$ yields higher computational efficiency. In a real-world setting, a continual learner has to adapt to a large-scale data stream (ideally never-ending), thereby requiring more computation than size of the data stream may not be feasible. For example, for many applications such as on-device learning, embedded devices, and AR/VR, a continual learner has to learn new information quickly without increasing computational overhead. Our principal focus was to align continual learning with resource-constrained applications. That’s why we chose $f$=0.3. We have revised the paper to better explain this choice. # Why Not Use ResNet Trained from Scratch? While ResNet-18 has been historically used, we see no reason to not use more recent architectures that would actually be employed in real-world applications. Moreover, ConvNeXtV2-Femto performs better than ResNet-18 (78.25% vs 69.76% on ImageNet-1K) using 2x fewer parameters (5.2M vs. 11.6M parameters) [1]. Many recent works are using vision transformers and more advanced architectures (e.g., [7]), and we believe that continual learning must use the latest advances for the community to care about it rather than focusing on the ResNet architecture which is now almost a decade old. Note that LoRA is not appropriate for ResNet architectures, but it can be used with ViT models, ConvNeXt, ConvNeXtV2, etc. Here are some additional reasons: 1. For the major real-world applications of continual learning, training from scratch is not employed. Hence, we focused our experiments on a setting that matches real-world use-cases, where companies and many academic projects start with ImageNet-1K pre-trained models. 2. Many recent methods in continual learning that have had significant impact use pre-trained models, e.g., DualPrompt uses a ViT pre-trained on ImageNet-1K. Indeed with the shift to ViT models, many continual learning papers are adopting this paradigm. We note that this is often hidden in their supplemental material rather than presented in the main text. We aim to be upfront about what we are doing. 3. Many continual learning methods evaluated only on ImageNet-1K have their first offline batch as the first 100-500 classes of ImageNet-1K [2-6], and we think that would suffice. For example, in ImageNet-1K experiments, BiC [5] and PODNet [6] used 500 ImageNet classes for initializing the network before the CL phase began on the remaining 500 ImageNet classes. During the initialization phase, they trained a ResNet-18 on 500 ImageNet Classes for 90 epochs. # Add Normal Fine-Tuning Results to Table 1 and ImageNet-1K Results Done. # Does Stability Take Precedence Over Plasticity? Stability and plasticity are both measured in all experiments and reducing both stability and plasticity gaps indicates that stability and plasticity is well balanced. SGM achieves this goal in all experiments where it increases both stability and plasticity (Fig. 5). # Analyze The Stability Gap for the Initial Batch of Places365-LT This is provided in Fig. 4. Is there something else desired? # Use Balanced Datasets and Training from Scratch Experiments We included training from scratch experiments in supplemental section F. As shown in Table 10, our method improved performance over vanilla. Besides imbalance dataset (Places365-LT), we have included experiments with balanced datasets (Places365-Standard) as well. The results with a blanched dataset are presented in Table 3, 9, 10, and 11. # Do the Results Generalize? While our training from scratch experiments demonstrate SGM is effective in that setting, it showed less benefit. We hypothesize this is probably due to CIFAR being inadequate for learning good representations before plasticity is reduced. We are planning to conduct additional experiments with recent continual learning methods using pretrained ViT models, although we are not sure if we will have sufficient time to do this during the rebuttal period. # References 1. Liu et. al., “A ConvNet for the 2020s”, In CVPR 2022. 2. Belouadah et. al., "IL2M: Class Incremental Learning With Dual Memory", In ICCV 2019. 3. Rebuffi et. al., "iCaRL: Incremental classifier and representation learning", In CVPR 2017. 4. M. Castro et. al., "End-to-End Incremental Learning", In ECCV 2018. 5. Hou et. al., "Learning a Unified Classifier Incrementally via Rebalancing", In CVPR 2019. 6. Douillard et. al., "PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning", In ECCV 2020. 7. Wang et al., "DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning", In ECCV 2022.
Summary
In this work, the authors propose to mitigate the issue of the stability gap for continual learning. To achieve that, the authors first illustrate two hypotheses for the reasons that result in a stability gap: 1) high loss values for new classes; and 2) excessive network plasticity. Moreover, to examine the hypotheses, the authors introduce three stability metrics and further propose SGM to mitigate the stability gap. Specifically, SGM combines weight initialization, dynamic soft targets, LoRA, and freezing of the output layer of old classes. In the experiments, SGM shows better accuracy and stability compared to prior methods on three benchmarks.
Strengths
1. It's good to see the author's efforts to address a recently proposed stability gap issue for continual learning 2. The writing is relatively clear and easy to follow 3. The proposed three metrics to measure the stability gap seem novel.
Weaknesses
1. The extremely relaxed setting for continual learning limits the contribution of the proposed method. For this setting, the continual learner has a strong pre-trained model and can access all prior source data for data replay(e.g., ImageNet-1K) which is unusual in continual learning. 1.1 Such a setting narrows the proposed hypothesis for continual learning with the pre-trained model. Meanwhile, the proposed techniques are specially designed for that. With the pre-trained model, some of the proposed techniques like weight initialization and LoRA are natural selections that are hard to be considered as contributions. 1.2 Although the authors also conduct experiments on CIFAR-10 for training from scratch, the performance is not convincing compared to Vanilla Rehearsal only. !.3 Such extremely relaxed settings still can not reach the accuracy of offline settings as shown in Table.1 2. Lack of comparisons with most related works on continual learning with pre-trained models. In the paper, the authors mainly compare the proposed method to DERpp and GDumb which is designed for training from scratch. Thus, comparisons with the most recent works on continual learning with pre-trained models are needed. R1: A Unified Continual Learning Framework with General Parameter-Efficient Tuning (ICCV’23) R2: Learning to prompt for continual learning. CVPR 2022 R3: Dualprompt: Complementary prompting for rehearsal-free continual learning. ECCV 2022 R4: Isolation and Impartial Aggregation: A Paradigm of Incremental Learning without Interference. AAAI 2023 3. Lack of ablation study to show the performance of partially combined techniques in SGM. In the Table.1, the authors list the ablation study of each component of SGM. Since the proposed SGM includes four different techniques, it's unclear what is the performance of partially combined techniques in SGM.
Questions
Questions regarding the Weakness above: 1. The extremely relaxed setting for continual learning limits the contribution of the proposed method. 1.1 Why do the authors use an unusual setting for continual learning (i.e., a combination of ImageNet-1K and Places365 with special CL ordering)? 1.2 It's better to conduct experiments in more general settings, for example, 1) with a pre-trained model: a combination of ImageNet-1K and CIFAR-100 or ImageNet-R 2) without a pre-trained model: Split CIFAR-10/CIFAR-100 with the comparison with SOTA methods. 3. Lack of ablation study to show the performance of partially combined techniques in SGM. For example, what is the performance if only combining weight initialization and LoRA? 4. It's unclear why improving the stability gap can help to improve the final performance. For example, will it also mitigate the forgetting and improve forward and backward transfer capacity? Minor questions: 1. "Batches" and "Rehearsal cycles " seem to have the same meaning but are used at different places on the paper. It's better to keep the concept consistent to avoid confusion. In addition, the concept of "Batches" in the paper is easy to get confused with the hyper-parameter "Batch size" for training models.
Rating
5: marginally below the acceptance threshold
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
2 fair
Contribution
2 fair
Ethics concerns
N/A
Addressing Claimed Weaknesses
Thank you for taking the time to review our paper. # Concerns Regarding Starting with a Pretrained Model Given that some of the methods the reviewer mentions start with large pre-trained ViT models (Learning to Prompt and DualPrompt), this concern is especially surprising. Here are three reasons why we do not think this is a valid concern: 1. For the major real-world applications of continual learning, training from scratch is not employed. Hence, we focused our experiments on a setting that matches real-world use-cases, where companies and many academic projects start with ImageNet-1K pre-trained models. 2. Many recent methods in continual learning that have had significant impact use pre-trained models, e.g., DualPrompt uses a ViT pre-trained on ImageNet-1K. Indeed with the shift to ViT models, many continual learning papers are adopting this paradigm. We note that this is often hidden in their supplemental material rather than presented in the main text. We aim to be upfront about what we are doing. 3. Many continual learning methods evaluated only on ImageNet-1K have their first offline batch as the first 100-500 classes of ImageNet-1K [2-7], and we think that would suffice. For example, in ImageNet-1K experiments, BiC [6] and PODNet [7] used 500 ImageNet classes for initializing the network before the CL phase began on the remaining 500 ImageNet classes. During the initialization phase, they trained a ResNet-18 on 500 ImageNet Classes for 90 epochs. We see little scientific value in using an ImageNet-1K pre-trained model on CIFAR-100, and Places is a much harder and more realistic dataset. Could the reviewer please justify why split CIFAR-100 would be better as done in DualPrompt over class incremental learning with Places? # Concerns Regarding the Amount of Data Stored The claim that we are using a larger buffer size compared to many past works for large-scale continual learning is false (see reason 2). We believe we adequately addressed this in the paper for three reasons: 1. We demonstrate that SGM is effective with LwF which is rehearsal free and does not use additional memory. 2. We conducted extensive experiments in Table 3, where we bounded the buffer size by 24K samples, which is only 0.8% of the total number of images in the two datasets (3M from the 1.2M in ImageNet-1K and 1.8M in Places). Prior work on class incremental learning with ImageNet has used 20K images (1.7% of the 1.2M in ImageNet-1K) [2-7]. So as a fraction of the total dataset, our constrained experiments are _more_ restrictive than most past works. 3. The unconstrained experiments we conducted in Table 1 enable us to focus on our scientific question, and for industrial applications the major concern is compute whereas storage is very cheap. # Concerns About The State-of-the-Art We are aiming to do science rather than chase benchmarks. SGM mitigates the stability issues and aligns with our hypotheses. In Table 1, SGM almost achieves the same accuracy (70.30%) as offline upper bound (70.69%), there exists only 0.39% difference. In Table 10, SGM improves performance over compared methods by absolute 3.43% (50K samples in buffer) and 3.45% (5K samples in buffer) in final accuracy. We did a fair comparison when we combined SGM with other existing CL algorithms e.g., DERpp, GDumb, REMIND, LwF which are widely used CL baselines. All compared methods used the same pre-trained model. We wanted to cover a variety of CL algorithms: 1) rehearsal based: GDumb, 2) rehearsal and knowledge distillation: DERpp, 3) non-rehearsal and knowledge distillation: LwF, 4) online CL: REMIND. Of course there are many more different CL methods but experimenting with all past CL methods is intractable. We want to demonstrate that our method mitigates the stability gap and increases computational efficiency in most common CL methods. # Try Every Possible Combination of the Mitigation Methods There are 16 possible configurations, and we did 6 of these. It is worth noting that it requires significant computation and time to assess the performance of a model while being trained on a large scale ImageNet dataset combined with another dataset. Is there a specific hypothesis the reviewer has that would justify this? # References 1. Liu et. al., “A ConvNet for the 2020s”, In CVPR 2022. 2. Belouadah et. al., "IL2M: Class Incremental Learning With Dual Memory", In ICCV 2019. 3. Hayes et. al., "Remind your neural network to prevent catastrophic forgetting", In ECCV 2020. 4. Rebuffi et. al., "iCaRL: Incremental classifier and representation learning", In CVPR 2017. 5. M. Castro et. al., "End-to-End Incremental Learning", In ECCV 2018. 6. Hou et. al., "Learning a Unified Classifier Incrementally via Rebalancing", In CVPR 2019. 7. Douillard et. al., "PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning", In ECCV 2020. 8. Wang et al., "DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning", In ECCV 2022.
Addressing Reviewer Questions
# Why Use ImageNet-1K followed by Places-365? As described in the introduction, our goal is to study CL in large scale problems which are more realistic and align with real-world applications. While others have not done Places-365 after learning ImageNet-1K, to be honest, we think it makes no sense from either a scientific or practical purpose to take an ImageNet-1K pre-trained model and then learn something like split CIFAR-100. It is like taking an incredibly powerful network to then learn a toy dataset. We do not see the scientific value. Could the reviewer please elaborate on why they think this would be better? Note that Places-365 is widely used as hard out-of-distribution dataset in papers regarding out-of-distribution detection with ImageNet-1K datasets due to the domain shift between Places-365 and ImageNet-1K [1], whereas the classes in CIFAR significantly overlap with ImageNet-1K (see Table F.1 in [2]). Our problem setting is harder than the ones suggested by the reviewer. # Experiments with Models Trained from Scratch See Supplemental Section F. # Why does SGM Improve Final Performance? We demonstrate that SGM helps achieve best accuracy using significantly less training steps than vanilla (Fig. 1 and Fig. 4) since SGM mitigates the stability gap (drop in old task). On the other hand, vanilla (without SGM) suffers from the stability gap and thus requires significantly more training steps to reach higher accuracy (Fig. 1 and Fig. 4). As shown in Fig. 5, SGM retains the model's old task performance (stability) and improves its new task performance (plasticity). It is possible that SGM also improves forward and backward transfer which is indicated by negative stability and plasticity gaps. For class incremental learning (dissimilar tasks), SGM did not fully achieve negative stability/ plasticity gap but in IID CL (similar tasks), SGM achieves negative stability gap (Table 4) which indicates knowledge transfer from new classes to old classes. # References 1. Ziwei Liu et. al., “Large-Scale Long-Tailed Recognition in an Open World”, In CVPR, 2019. 2. Kornblith et. al., "Why do better loss functions lead to less transferable features?", In NeurIPS, 2021
Summary
The paper proposes a new method to reduce the stability gap, which is a phenomenon that occurs in continual learning when learning new data, where accuracy on previously learned data drops significantly before recovering. The proposed method, Rehearsal with Stability Gap Mitigation (RSGM), consists of 5 different techniques: Weight Initialization, Dynamic Soft Targets, old output class freezing (OOCF) and LoRA. The experiment results show that the proposed method reduces the stability gap in class incremental learning and reduces the number of network updates.
Strengths
1. Stability gap phenomenon is a recently discovered. Mitigating it can play a crucial role in enhancing the efficiency of continual learning. 2. It propose new metrics, which differ slightly from those proposed by De Lange et al., to measure the stability gap in class incremental learning. 3. It offers extensive ablation studies to facilitate a comprehensive understanding of the effect on each compoenent.
Weaknesses
1. The proposed method is evaluated in constrained settings. The network is ConvNeXtV1-Tiny, which is seldomly used in previous continual learning literature. The applicability of the proposed method to other network architectures, such as ResNet or simple CNNs, remains unclear. Additionally, the buffer size is much larger than the one used in previous literature. 2. The effectiveness in reducing the stability gap depends heavily on the utilization of a pretraining model from ImageNet-1K. As shown in Tab. 10, the improvement is very limited when training from scratch, due to the fact that LoRA and freezing of old class units cannot be used in this case.
Questions
1. It is interesting that though the main goal of the proposed method is not to improve performance, it performs slightly better than the vanilla method. Can you explain this? 2. Can you explain why the metric in (De Lange et al. 2023) is model-dependent and cannot be used to compare different approaches? In (De Lange et al. 2023). Please provide further elaboration on why the proposed new metrics are considered superior to the metrics presented in (De Lange et al. 2023). 3. What does the ‘Best Mitigation Method’ shown in Figure 1 refer to? Is it the proposed RSGM? A suggestion: the term "Offline" may cause some confusion, despite the clarification in the main text. It would be clearer to use "jointly" training, instead as "offline" is commonly used to distinguish between online and offline continual learning.
Rating
5: marginally below the acceptance threshold
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Addressing the Claimed Weaknesses
Thank you for taking the time to review our paper. # Concerns Regarding ConvNext While ResNet-18 has been historically used, we see no reason to not use more recent architectures that would actually be employed in real-world applications. Moreover, ConvNeXtV2-Femto performs better than ResNet-18 (78.25% vs 69.76% on ImageNet-1K) using 2x fewer parameters (5.2M vs. 11.6M parameters) [1]. Many recent works are using vision transformers and more advanced architectures (e.g., [8]), and we believe that continual learning must use the latest advances for the community to care about it rather than focusing on the ResNet architecture which is now almost a decade old. Note that LoRA is not appropriate for ResNet architectures, but it can be used with ViT models, ConvNeXt, ConvNeXtV2, etc. # Concerns Regarding the Amount of Data Stored The claim that we are using a larger buffer size compared to many past works for large-scale continual learning is false (see reason 2). We believe we adequately addressed this in the paper for three reasons: 1. We demonstrate that SGM is effective with LwF which is rehearsal free and does not use additional memory. 2. We conducted extensive experiments in Table 3, where we bounded the buffer size by 24K samples, which is only 0.8% of the total number of images in the two datasets (3M from the 1.2M in ImageNet-1K and 1.8M in Places). Prior work on class incremental learning with ImageNet has used 20K images (1.7% of the 1.2M in ImageNet-1K) [2-7]. So as a fraction of the total dataset, our constrained experiments are _more_ restrictive than most past works. 3. The unconstrained experiments we conducted in Table 1 enable us to focus on our scientific question, and for industrial applications the major concern is compute whereas storage is very cheap. # Concerns Regarding Starting with a Pretrained Model We disagree that this is a concern for three reasons: 1. For the major real-world applications of continual learning, training from scratch is not employed. Hence, we focused our experiments on a setting that matches real-world use-cases, where companies and many academic projects start with ImageNet-1K pre-trained models. 2. Many recent methods in continual learning that have had significant impact use pre-trained models, e.g., DualPrompt uses a ViT pre-trained on ImageNet-1K. Indeed with the shift to ViT models, many continual learning papers are adopting this paradigm. We note that this is often hidden in their supplemental material rather than presented in the main text. We aim to be upfront about what we are doing. 3. Many continual learning methods evaluated only on ImageNet-1K have their first offline batch as the first 100-500 classes of ImageNet-1K [2-7], and we think that would suffice. For example, in ImageNet-1K experiments, BiC [6] and PODNet [7] used 500 ImageNet classes for initializing the network before the CL phase began on the remaining 500 ImageNet classes. During the initialization phase, they trained a ResNet-18 on 500 ImageNet Classes for 90 epochs. # Concerns Regarding Our Training From Scratch Experiments In Table 10, our training from scratch experiments with CIFAR show that SGM improves performance over compared methods by absolute 3.43% (50K samples in buffer) and 3.45% (5K samples in buffer) in final accuracy. Regardless, we think this benefit would be much more dramatic if the network was trained with a larger dataset, as there isn’t much to learn in terms of strong representations for such a toy problem. # Questions for the Reviewer We don't know if we would have adequate time, but we are considering these: 1. Would experiments using SGM with the common paradigm of pre-training on the first 100 classes of ImageNet-1K followed by class incremental learning of the remaining 900 classes shift their opinion? 2. Would experiments involving training a model on low resolution data, e.g., pre-training on CIFAR-100, and then learning a sequence of new low resolution datasets, e.g., CIFAR-10 (which has no overlapping classes) and then down-scaled versions of higher resolution datasets be adequate evidence? 3. Would analyzing SGM's behavior on a popular method such as DualPrompt using their paradigm be sufficient evidence? # References 1. Liu et. al., “A ConvNet for the 2020s”, In CVPR 2022. 2. Belouadah et. al., "IL2M: Class Incremental Learning With Dual Memory", In ICCV 2019. 3. Hayes et. al., "Remind your neural network to prevent catastrophic forgetting", In ECCV 2020. 4. Rebuffi et. al., "iCaRL: Incremental classifier and representation learning", In CVPR 2017. 5. M. Castro et. al., "End-to-End Incremental Learning", In ECCV 2018. 6. Hou et. al., "Learning a Unified Classifier Incrementally via Rebalancing", In CVPR 2019. 7. Douillard et. al., "PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning", In ECCV 2020. 8. Wang et al., "DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning", In ECCV 2022.
Addressing the Questions Posed
# Why Does SGM Perform Better than Vanilla? We hypothesize that the mechanisms in SGM facilitate better forward and backward transfer, resulting in better performance than the vanilla method. That said, our primary objective is to study the stability gap in order to reduce the amount of compute needed to update models trained with rehearsal, which is the method that holds the most promise for industrial applications, where compute is limited, storage is cheap, and predictive performance cannot be sacrificed. The goal of SGM is to reduce stability gap, plasticity gap and continual knowledge gap and enhance performance while using fewer network updates as shown in Fig.1-5. # What's Wrong With the Metrics in De Lange et al. (2023)? The metrics in De Lange et al. (2023) are not normalized and focus on worst-case performance during continual evaluation of a model over a sequence of tasks. This means it cannot be used to analyze whether the model is meeting the needs of industry to catch-up to the offline upper bound. By normalizing our scores with a universal offline model trained jointly on all of the data trained from scratch we can then compare across approaches. Specifically, their worst-case evaluation attempts to find the largest drop relative to the same model’s best performance, which may be quite far from an offline model. Instead, our metrics measure performance compared to a much stronger upper bound. Would it be helpful to the reviewer if we also provided their metrics in supplemental material? If so, we could potentially do this. # What is the "Best Mitigation Method" in Figure 1? The best mitigation method is our proposed method, SGM. We have revised the caption and figure to be more specific.
Global Response to Reviewers
We thank all of the reviewers for spending time reviewing our manuscript. We want to address some common concerns. # Justification for Our Setting Multiple reviewers had concerns about our setting differing from those of others. We firmly believe that for continual learning to have real-world impact, we need to align research with real-world needs and the rest of the deep learning community, rather than persevering with settings that make little sense from a scientific or practical standpoint. Specifically these are the concerns: 1. **Use of Pre-trained Models** - Pretrained models are widely used throughout deep learning and industry, and the major real-world use-case for continual learning is keeping models updated. Moreover, there are a growing number of continual learning papers, some of which reviewers cited, that are using large pre-trained ViT models that have been pre-trained on ImageNet-1K or ImageNet-21K [3 - 7]. Training from scratch does not help achieve computational efficiency needed for real-world applications. Moreover, many continual learning methods that are trained from scratch fail on real-world datasets [8 - 10]. 2. **Justification for Using Places-365** - While others are using pre-trained ViT models, many of their experiments apply them to toy datasets such as CIFAR. This is taking an incredibly powerful network and then applying it to a dataset where the primary source of domain shift is due to image resolution rather than having inputs that differ greatly in terms of semantics. Instead, we used Places-365, which is widely used as a challenging out-of-distribution recognition dataset in experiments involving ImageNet-1K pre-trained networks. Given the _significant_ overlap between CIFAR-10/100 and ImageNet-1K (see Table F.1 in [2]), we think there needs to be a sound scientific reason for using CIFAR-10/100 instead of Places-365. 3. **Training From Scratch** - We note that we did this in the paper; however, this was restricted to a toy dataset. 4. **Not using ResNet** - ResNet is almost a decade old and we cannot apply LoRA to it. We are using a modern performant architecture that has higher accuracy with half the parameter count of ResNet-18. # Why We Are Doing Science Instead of Chasing the SoTA The paper that introduced the stability gap [1], considered most basic CL baselines e.g., experience replay, GEM, EWC, SI, and LwF. None of these methods are SoTA. Their pursuit was scientific analysis where they showed that the stability gap is present in various _types_ of continual learning algorithms: rehearsal (GEM, experience replay), regularization (EWC, SI), and knowledge distillation (LwF). Following, that research program, we performed hypothesis-driven analysis to mitigate the stability gap. We showed that our method mitigates the stability gap and enables greater computational efficiency across rehearsal (vanilla, GDumb), non-rehearsal (LwF), knowledge distillation (DERpp), and online learning (REMIND) methods. # Concerns Regarding Memory Constraints We are confused by comments from reviewers regarding the memory constraints we imposed as being excessively generous in our constrained memory experiments. We bounded the buffer size by 24K samples, which is only 0.8% of the total number of images in the two datasets (3M from the 1.2M in ImageNet-1K and 1.8M in Places). Prior work on class incremental learning with ImageNet-1K has used 20K images (1.7% of the 1.2M in ImageNet-1K) [3 - 7]. So as a fraction of the total dataset, our constrained experiments are _more_ restrictive. We also conducted experiments without memory constraints to both remove that variable from analysis to understand the effectiveness of our methods, and because it aligns with industry's interest in continual learning, where compute is restrictive but they have access to all of their data. Many in industry are asking the continual learning community to address this. # References 1. De Lange et. al., "Continual evaluation for lifelong learning: Identifying the stability gap", In ICLR 2023 2. Kornblith et. al., "Why do better loss functions lead to less transferable features?", In NeurIPS 2021 3. Gao et. al., "A Unified Continual Learning Framework with General Parameter-Efficient Tuning", In ICCV 2023 4. Wang et. al., "DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning", In ECCV 2022 5. Castro et. al., "End-to-End Incremental Learning", In ECCV 2018 6. Hou et. al., "Learning a Unified Classifier Incrementally via Rebalancing", In CVPR 2019 7. Douillard et. al., "PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning", In ECCV 2020 8. Ramasesh et. al., "Effect of Model and Pretraining Scale on Catastrophic Forgetting in Neural Networks", In ICLR 2022 9. Mehta et. al., "An Empirical Investigation of the Role of Pre-training in Lifelong Learning", In ICML 2021 10. Lee et. al., "Do Pre-trained Models Benefit Equally in Continual Learning?", In WACV 2023
Continual Learning for Real-World Applications
If reviewers are interested in learning about what's needed for real-world continual learning and the primary application, this blog post is very insightful: https://huyenchip.com/2022/01/02/real-time-machine-learning-challenges-and-solutions.html#stateful-training See the section about Stateful Training on keeping production AI systems up-to-date by training them continuously. This is the major application of interest in industry for continual learning, and storage is primarily not a concern at all in that setting.
Proposed Experiments
To address some of the reviewers concerns, we are considering conducting the following experiments, although given the limited amount of time we want to ensure that any would be of sufficient interest to them and would adequately address their concerns. 1. Training a ViT model on CIFAR-100 and then doing continual learning on CIFAR-10, which contain mutually exclusive classes. 2. Analyzing SGM's impact on an ImageNet-1K pre-trained continual learning method such as DualPrompt and then replicating their experiments (e.g., using split CIFAR-100) or using Places-365. 3. While we cannot apply LoRA to ResNet-18 because it lacks the linear layers of ConvNeXt, we could apply everything else besides LoRA that is in SGM to ResNet-18. We could then study its behavior in regards to class incremental learning on some dataset like CIFAR-10 or Places-365. Please let us know what you think, although time is quite limited during the rebuttal period. We do intend to study something like option 2, regardless, but given other deadlines we may not prioritize this until later unless it is endorsed by our reviewers as being of sufficient value for the rebuttal.
New revision submitted
Dear Reviewers, Thank you for your comments and for the time and effort you put into assessing our paper. We appreciate your constructive feedback, valuable insights, and questions. We carefully considered all comments provided by reviewers and addressed all of them appropriately. We provided detailed responses to each review separately and we believe that our responses addressed all of the reviewers' concerns. We have made following changes in the revised manuscript to address reviewers’ concerns: 1) We have included two more baselines in Table 1. They are: naive finetune (lower bound) and head (only output layer is trainable). We have also explicitly reported ImageNet-1K accuracy in Table 1 due to class imbalance. 2) We have updated Fig.3 with FLOPs in addition to the number of network updates. It is evident that SGM provides computational benefits over vanilla rehearsal. We have also updated Fig.1 to clarify that our method, SGM, achieves 16.7x speedup. 3) Appendix J includes additional computational efficiency comparisons. All results confirm that SGM achieves significant computational efficiency. 4) Since several reviewers asked for comparison with DualPrompt, we have conducted new experiments and summarized our findings in Appendix K. We find that SGM enhances DualPrompt’s performance on both CIFAR-100 and ImageNet-R datasets. 5) To address concerns regarding our chosen experimental settings, we have included a discussion in Appendix L. We have tried to justify our design choices for datasets, architectures, pretrained models, memory and compute constraints, and new metrics. 7) Appendix C includes weight initialization details. 8) We have also adjusted texts throughout the paper according to reviewers’ comments. In summary, our proposed method, SGM works well across multiple networks (ConvNeXt V1, ConvNeXt V2, MobileViT, ViT), multiple datasets (ImageNet-1K, Places365-LT, Places365-Standard, CUB-200, CIFAR-10, CIFAR-100, ImageNet-R), multiple CL settings (class incremental, continual IID, online learning, rehearsal-free), different CL methods (vanilla/ experience replay, DERpp, GDumb, REMIND, LwF, DualPrompt), and constraints (compute only, both compute and memory), demonstrating its broad applicability. NB. We have attached the appendices (supplementary materials) with the main paper. Please discard the previous separate file for the supplementary materials. Kind regards, the authors
Meta-review
This paper focuses on an issue related to catastrophic forgetting where, at the point of updating with new data, performance on old data temporarily and dramatically drops, referred to as the stability gap. The work provides extensive experiments on a proposed approach consisting of a number of heuristic-based techniques. Strengths: - The experiments contain many datasets, different models, and many baseline methods. Weaknesses: - While the proposed approach shows improved performance, the approach contains some heuristics that are undermotivated. For example, the soft target idea is an intuition-based modification to softmax classification. While the latter is a well-understood principled approach, no theoretical analysis or insight into the validity or correctness of the proposed soft-target approach is given. The same objection applies to the proposed approach of targeted freezing. - As reviewers pointed out, LoRA is a natural choice in the proposed setup with pretrained network and cannot be considered novel contributions. - The reviewers brought concern regarding the choice of setup with pretrained network and its implication on the claims made on the experimental results. While I agree with the authors that pretraining setup is common, it is clear that other setups are also seen as similarly important by some of our colleagues. The implication of this choice is that the scope of the work is narrower than it seems. By itself, it is not an issue, but it still requires acknowledgment and careful framing on the authors’ part. - While the paper compares computational efficiency in terms of the number of iterations required to overcome the stability gap, it does not address common measures of computational overhead, such as complexity and training time. I notice a significant point of conflict between authors and reviewers regarding the scope of the work and the perception of what is considered an acceptable setup or baseline. I highly recommend clarifying the setups and justifying the choices more carefully in future submissions.
Why not a higher score
While I think the work is highly interesting with significant results, the points of conflict between the authors and reviewers, some of which I agree with and listed as weaknesses, are not resolved. This paper could have been recommended for acceptance if we had an agreement on the authors’ part to clarify some of the confusion. Unfortunately, we did not even come close to such an agreement.
Why not a lower score
N/A