SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Models

Further Clarifications (1/3)

Thank you for taking the time to review our previous response and provide your feedback. We appreciate your concerns and would like to further **clarify the effectiveness of our work from the following four perspectives** for your consideration. **A. Significant Overall Performance of Our Method** First of all, as validated in Table 1 of our paper, compared to recent SOTAs, **our method shows substantial improvements across all 6 popular datasets**. For example, we surpass the second-best result on ImageNet-A over **4%** which indicates notable superiority. Second, we would like to further highlight the **significant relative improvements** against previous methods. In particular, we compare our method with two representative SOTA methods, RanPAC [1] and SSIAT [2], from NeurIPS2023 and CVPR2024 respectively, and results are summarized in the table below. We remark that 1) SSIAT [2] did not report (denoted by symbol "-") the result on Omnibenchmark (OB) and hence we compute both the average of all available datasets except OB and the average of all 6 datasets; 2) Metrics in the brackets denotes the corresponding increase (↑) / decrease (↓) compared to its previous SOTA. | Method | CIFAR | IN-R | IN-A | CUB | VTAB | OB | Avg of first 5 datasets (Avg5) | Avg of all 6 datasets (Avg6) | |----------------------|-------|-------------|-------------|-------------|-------------|------|-------------|-------------| | RanPAC [1] (NeurIPS 2023) | 92.2 | 78.1 | 61.8 | 90.3 | 92.6 | 79.9 | 83.0 | 82.5 | | SSIAT [2] (CVPR 2024) | 91.4 (↓0.8) | 79.6 (↑1.5) | 62.2 (↑0.4) | 88.8 (↓1.5) | 94.5 (↑1.9) | - | 83.3 (↑0.3) | - | | **Ours** | **92.8 (↑1.4)** | **81.0 (↑1.4)** | **66.6 (↑4.4)** | **91.1 (↑2.3)** | **95.0 (↑0.5)** | **80.9 (↑1.0)** | **85.3 (↑2.0)** | **84.6 (↑2.1)** | One can observe from the table that: - **The relative improvements of our method compare favorably against those of the most recent SOTA SSIAT [2]**. Instead of only 0.3% increase of SSIAT [2] in Avg5 against its previous SOTA RanPAC [1], our method boosts the metric by a significant margin 2.0% compared to SSIAT [2] and outperforms RanPAC [1] by 2.1% in Avg6 metric. - Our method shows **consistent improvement over all datasets**. Compared to SSIAT [2] even performs inferior to its previous SOTA RanPAC [1] on CIFAR and CUB, our method outperforms previous methods on all the datasets with various characteristics. This validates the robustness of the proposed method. Based on the above observations, we believe that our method is sufficiently effective when compared with baselines. In addition, we would like to remind the reviewer that due to the **inherent difficulty of the tasks**, the trade-off between stability and plasticity, and the fact that our method builds upon pre-trained models (PTMs) rather than starting from scratch, these improvements are even more challenging to achieve. **Other methods show less significant improvements** [3][5]. For instance, **EASE** [5], accepted by **CVPR 2024**, achieved a **0.03%** improvement in final accuracy and a **0.08%** increase in average accuracy on CUB compared to the second-best result. On OmniBenchmark, EASE outperformed the second-best result by **0.06%**. **FeCAM** [3], accepted by **NeurIPS 2023** did **not** surpass recent exemplar-based state-of-the-art results on CIFAR. [1] RanPAC: Random Projections and Pre-trained Models for Continual Learning. NeurIPS2023. [2] Semantically-Shifted Incremental Adapter-Tuning is A Continual ViTransformer. CVPR2024. [3] Exploiting the heterogeneity of class distributions in exemplar-free continual learning. NeurIPS2023. [4] SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model [5] Expandable subspace ensemble for pre-trained model-based class-incremental learning. CVPR2024. [6] A Unified Continual Learning Framework with General Parameter-Efficient Tuning. ICCV2023.

Further Clarifications (2/3)

**B. Notable Component-Wise Improvement** Our method is composed of three key components: a slow learner, a fast learner, and an aggregation module. We would like to emphasize that **each component plays a significant role in the overall effectiveness of our approach**. 1) **Slow Learner**: Our method demonstrates **significant improvements over the baseline and other alternatives**: **1)** As shown in Table 2 of the manuscript, the proposed slow learner surpasses the baseline method RanPAC [1] by **3.24%** on ImageNet-A. **2)** As indicated in Table 7 of the rebuttal PDF, the proposed slow learner achieves an **average improvement of 1.19%** in final accuracy **across six datasets** compared to the baseline. **3)** As shown in Table 4 of the manuscript, the slow learner **outperforms competing methods** that also aim to transfer statistical knowledge in PTMs. The final accuracy of the slow learner improves by **2.63%** compared to first-order information transfer and by **2.24%** compared to second-order information transfer. **4)** The visualization via t-SNE shows **superior generalization** of the slow learner **on unseen classes** in Figure 3 in the manuscript. The key results are summarized in the following table. | Method | Performance (%) | Competitor | Data Source | |:------------:|:---------------:|:---------------------------------:|:-----------:| | Slow learner | +3.24 | Baseline | Table 2 | | Slow learner | +2.63 | First-order information transfer | Table 4 | | Slow learner | +2.24 | Second-order information transfer | Table 4 | 2) **Fast Learner**: Our method shows **notable improvements compared to the baseline and other approaches**. Specifically: **1)** As shown in Table 2 of the manuscript, the proposed fast learner **surpasses the baseline by 4.29%** on ImageNet-A. **2)** As indicated in Table 7 of the submitted PDF, the proposed fast learner achieves an **average improvement of 1.72%** in final accuracy **across six datasets** compared to the baseline. **3)** The following ablation table, which extends Table 5 of the manuscript with a new row, demonstrates that **both $\mathcal{L}\_{\mathrm{cos}}$, $\mathcal{L}\_{\mathrm{f2s/s2f}}$ contribute significant improvements.** Specifically, solely using **$\mathcal{L}\_{\mathrm{f2s/s2f}}$** results in an improvement of **3.10%** compared to the baseline, while **$\mathcal{L}\_{\mathrm{cos}}$** yields a gain of **3.86%** over the baseline. **4)** As shown in Table 11 of the rebuttal PDF, the fast learner guided by the slow learner achieves an improvement in final accuracy of **10.73%** and **2.83%** compared to **other teacher choices**. **5)** As shown in Figure 7(b) in the additional PDF and Figure 4 in the submitted manuscript, **fast learner compensates for the plasticity** issue of slow learner and demonstrates superiority in new classes. | Method | Final | Avg | | --------------------------------------------------------------------------------- | --------- | --------- | | Baseline | 62.21 | 72.31 | | Fast Learner w/o $\mathcal{L}\_{\mathrm{f2s/s2f}}$, $\mathcal{L}\_{\mathrm{cos}}$ | 8.16 | 30.73 | | Fast Learner w/o $\mathcal{L}\_{\mathrm{cos}}$ | 65.31 | 73.88 | | Fast Learner w/o $\mathcal{L}\_{\mathrm{f2s/s2f}}$ | 66.07 | 74.20 | | **Fast Learner** | **66.49** | **74.50** | 3) **Aggregation Module**: - **First**, the proposed aggregation method demonstrates **significant improvements over the baseline and other alternatives**. Specifically: **1)** As indicated in Table 7 of the rebuttal PDF, the proposed aggregation approach achieves an **average improvement of 2.1%** in final accuracy **across six datasets** compared to the baseline. **2)** As shown in Table 3 of the submitted manuscript, the aggregation method results in **at least a 0.53% improvement** in final accuracy compared to straightforward baselines. - **Second**, the proposed aggregation method **does not require complex modules or trainable parameters**, making it straightforward to implement. - **Third**, it is important to recognize that **aggregate** methods are generally more **challenging to improve**. For example, the **LAE**[6] method from **ICCV 2023** achieved only a **0.4%** increase in average accuracy, with a **0.4% decrease** in final accuracy on ImageNet-R. In contrast, our method demonstrates a **2.85% final** improvement on ImageNet-R and an **average** improvement of **2.26%**.

Further Clarifications (3/3)

**C. Unified Framework Synergy** We would like to emphasize that our approach serves as **a unified framework** for PTM-based continual learning. Specifically, we would like to clarify that: 1) The effectiveness of our framework lies not only in the contribution of each individual component but also in the overall improvement it delivers, which can make the **impact of each component less apparent** when viewed in isolation. A similar situation is **observed in RanPAC** [1], accepted by **NeurIPS 2023**, where Phase 1 (a single component) results in improvements of only **0.3%** on CUB and OmniBenchmark, and **0.2%** on VTAB. In contrast, our two learners achieve a minimum improvement of **0.36%** on CUB, **0.75%** on OmniBenchmark, and **1.11%** on VTAB. **Furthermore**, our method consistently demonstrates **substantial improvements across all six popular datasets**. 2) The **components** of our method are **interdependent** and should not be separated. For example, the **fast learner relies on the slow learner for guidance** during adaptation. As demonstrated in Table 11 of the rebuttal PDF, the performance of the fast learner degrades by **10.73%** and **2.83%** when guided by alternative teachers. **D. Contributions Beyond Experimental Results** We would like to draw the reviewer's attention to the **unique contributions and features** of our work (particularly the **theoretical** foundations that set our work apart): 1) To the best of our knowledge, we are the first to apply **slow and fast learning to parameter-efficient tuning** for addressing challenges in PTM-based continual learning. 2) Our slow learner inherits generalizable knowledge from PTMs, supported by **theoretical analysis** using information bottleneck theory, a crucial aspect often **overlooked in previous works**. This transfer enables the slow learner to generalize well to novel classes that it has not encountered during training. 3) We address the **current limitations in adaptation** by guiding the fast learner with the slow learner, **eliminating the need for access to data or data distributions** [2][4]. During this process, the proposed fast learner continuously adapts to new knowledge while effectively mitigating the forgetting of previous knowledge. --- In conclusion, we believe that the thorough analysis and evidence provided in our response demonstrate the robustness and effectiveness of our proposed method. By addressing each of the reviewer's concerns and presenting detailed comparisons with SOTA approaches, we have shown that our work not only advances the field but also offers unique contributions through its theoretical foundations and innovative components. **We sincerely hope that this additional clarification helps to illustrate the significance of our research and leads to a favorable evaluation.** Thank you once again for your thoughtful review and consideration.

Further Clarifications

Thank you for your feedback. We greatly appreciate the time and effort you have put into reviewing our submission. We have carefully considered your concerns and have endeavored to address them in our rebuttal. Below, we would like to summarize the key points regarding your raised concerns. 1) Regarding the concerns about training the slow adapter only in the initial session, we have provided both qualitative and quantitative explanations to demonstrate its effectiveness and necessity compared to periodic updates. 2) For concerns related to loss functions and hyper-parameters, we offer detailed ablations and analyses to highlight their specific contributions and to show the stability of the default hyper-parameter choices, which facilitate practical implementation. 3) Additionally, we provided a specific explanation in the manuscript for why the proposed method was not compared with O-LoRA, an inspiring work in continual learning. Thank you for bringing this to our attention; **we will include O-LoRA** in the Related Work section of the **revised version** for discussion and comparison. Moreover, we would like to draw the reviewer's attention to the **unique contributions and features** of our work, particularly the **theoretical foundations** that set our work apart. 1. To the best of our knowledge, we are the first to apply **slow and fast learning to parameter-efficient tuning** in addressing challenges in PTM-based continual learning. 2. Our slow learner inherits generalizable knowledge from PTMs, underpinned by **theoretical analysis** using information bottleneck theory—a crucial aspect often **overlooked in previous works**. This transfer allows the slow learner to generalize effectively to novel classes it has not encountered during training. 3. We tackle the current limitations in adaptation by guiding the fast learner with the slow learner, thereby **eliminating the need for access to data or data distributions**. Throughout this process, the proposed fast learner continuously adapts to new knowledge while effectively mitigating the forgetting of previous knowledge. 4. The clear demonstration of the effectiveness and superiority of our proposed method. We hope these additional details and clarifications address your concerns. **We welcome any further discussion** to enhance the clarity and quality of our work. Considering the extensive elaborations and the unique contributions of our research, **we would like to humbly request a reconsideration of the scoring**. We believe our research offers valuable insights and contributions that would be of significant interest to the NeurIPS community. Thank you again for your time and thoughtful consideration.

Further Clarifications

Thank you for your thoughtful feedback. We greatly appreciate the time and effort you have put into reviewing our submission. We have carefully considered your concerns and have endeavored to address them in our rebuttal. Below, we would like to summarize the key points regarding your raised concerns. 1. We provide further explanations to address your misunderstanding that the slow learner operates under certain assumptions. We also experimentally verified that constraining the fast learner to the slow learner yields better results. 2. We apply the proposed method to datasets with significantly varying data distributions between tasks, such as VTAB and DomainNet, and achieve state-of-the-art results. This demonstrates that our method is effective in scenarios where the data distribution of the first task diverges significantly from that of subsequent tasks. 3. For the concerns related to loss functions and hyper-parameters, we offer additional ablations and details to highlight the stability of the default choices, which are straightforward to implement in practice. 4. Thank you for bringing DAP to our attention. **We will include a discussion and comparison with it in the Related Work** of the revised version, further demonstrating the superiority of our approach. 5. Regarding concerns about the improvements of each component, we provide additional experimental results to demonstrate the significant improvement of each component in the overall effectiveness of our approach and their collaborative impact. In addition, we would like to draw the reviewer's attention to the **unique contributions and features** of our work, particularly the **theoretical foundations** that distinguish it. 1. To the best of our knowledge, we are the first to apply **slow and fast learning to parameter-efficient tuning** in addressing challenges in PTM-based continual learning. 2. Our slow learner inherits generalizable knowledge from PTMs, underpinned by **theoretical analysis** using information bottleneck theory—a crucial aspect often **overlooked in previous works**. This transfer allows the slow learner to generalize effectively to novel classes it has not encountered during training. 3. We tackle the current limitations in adaptation by guiding the fast learner with the slow learner, thereby **eliminating the need for access to data or data distributions**. Throughout this process, the proposed fast learner continuously adapts to new knowledge while effectively mitigating the forgetting of previous knowledge. 4. We provide extensive experiments on multiple benchmarks, providing strong evidence for the effectiveness of the proposed method. We hope these additional details and clarifications address your concerns. **We remain open to further discussion** to ensure the clarity and quality of our work. Given the extensive elaborations and the unique contributions of our research, **we respectfully request a reconsideration of the score**. We believe our work offers valuable insights that would be of great interest to the NeurIPS community. Thank you again for your time and consideration.