Mixture of Experts Meets Prompt-Based Continual Learning

Summary

The paper titled "Mixture of Experts Meets Prompt-Based Continual Learning" explores the integration of prompt-based continual learning methods with mixture of experts (MoE) architectures. The paper proposes a novel gating mechanism called Non-linear Residual Gates (NoRGa) to enhance the performance of prompt-based continual learning by leveraging theoretical insights and empirical evidence.

Strengths

1. This paper offers a novel connection between prompt-based-tuning and mixture-of-experts, providing a fresh perspective on prompt-based continual learning approaches. 2. Introduction of NoRGa, which integrates non-linear activation and residual connections to enhance continual learning performance while maintaining parameter efficiency.

Weaknesses

1. Comparing NoRGa with other state-of-the-art continual learning methods that do not use prompts would highlight the specific advantages of the proposed method. 2. It may be better to use a additional graph to represent the final method NoRGa of the paper. 3. It may be necessary to further clarify the differences between the proposed method and other prompt-based methods.

Questions

Which dataset is the ViT-B/16 pre-trained on.

Rating

6

Confidence

2

Soundness

3

Presentation

2

Contribution

3

Limitations

This paper has already discussed the limitations of our work in the conclusion section.

Summary

The paper explores the theoretical underpinnings and practical implications of prompt-based methods in continual learning, aiming to enhance our understanding and optimize their effectiveness. It introduces a novel perspective by connecting prefix tuning with mixture of experts models, revealing insights into how self-attention mechanisms encode specialized architectures. The proposed Non-linear Residual Gates (NoRGa) further advances this by improving within-task prediction accuracy and overall continual learning performance while maintaining parameter efficiency.

Strengths

1. This paper introduces a novel theoretical framework connecting self-attention mechanisms to mixture of experts models, significantly advancing the understanding of prompt-based approaches in continual learning. 2. The theoretical insights are well-supported and complemented by empirical experiments across diverse benchmarks, demonstrating robustness and reliability. 3. The concepts, although complex, are explained with clarity, aided by concrete examples and theoretical justifications. 4. Addressing the gap in theoretical understanding, the paper proposes a practical enhancement (NoRGa) that promises substantial improvements in model adaptation and efficiency.

Weaknesses

1. The paper could benefit from clearer explanations regarding the core elements of prompt-based methods, particularly in how prompts are defined and utilized within their framework. 2. While NoRGa is presented as a novel gating mechanism, its explicit connection to prompts could be elaborated further. Is it reasonable to interpret NoRGa's gating mechanism as a variant or extension of prompts in a broader sense?

Questions

1. Could NoRGa's gating mechanism be seen as a different interpretation or implementation of prompts, albeit operating at a different level or with distinct functional goals? 2. Are there instances of redundant theoretical explanations or definitions throughout the paper? Streamlining these could improve clarity and focus on the novel contributions without detracting from the foundational concepts. 3. Given the complexity of the concepts, would integrating more visual aids (e.g., diagrams illustrating the architecture of NoRGa or the relationship between prompts and gating mechanisms) enhance the paper's accessibility and readability?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper's reliance on dense theoretical explanations without adequate visual aids may limit its accessibility. Visual representations could significantly enhance comprehension of complex concepts like the relationship between prompts and NoRGa, potentially broadening the paper's audience and impact.

Summary

This paper introduces an extension to prefix tuning, by introducing non-linear residual gating - a simple extension over existing prefix tuning. This non-linear residual gating is supported with theoretical efficiency guarantees (polynomial versus exponential) to better estimate the optimal parameters. When applied to the problem of rehearsal-free continual learning with pretrained models (over four standard benchmarks) on top of the HiDe-Prompt method, the suggested gating provides consistent and partly significant improvements.

Strengths

* To the best of my knowledge in both the Continual Learning and the parameter-efficient finetuning domain, the non-linear residual gating prefix MoE is a novel extension over standard prefix tuning. * This paper is generally well written, and except for the proofs, quite easy to follow. * Assuming the regression estimator to be a fair assumption, the statistical sample efficiency improvements of the proposed NoRGa setup versus normal gating is significant (polynomial versus exponential). * The explicit improvements over HiDE-Prompt are consistent, and often significant.

Weaknesses

My biggest issue with this work is its placement as a Continual Learning contribution, and evaluating it as such. __[1]__ Firstly, the proposed approach, whilel helping with WTP, is simply an extension on top of prefix tuning - and none of the contributions connect to the continual distribution shift nature. The improved performance is much more likely tied to an improvement in the underlying PEFT approach; which in itself is simply sufficient on standard benchmarks (see Zhou et al. 2023). This is further supported by the fact that the authors freeze both alpha and tau after the first task (see supplementary D). Moreover, on such small benchmarks, introducing two additional tunable hyperparameters does incorporate additional degrees of freedom to overfit to these benchmarks, and makes direct comparison to existing methods difficult. Consequently, the question needs to be answered: How robust is NoRGa with respect to the additional introduced alpha and temperature? These are strictly two hyperparameters more than HiDe-Prompt. As such, how much more compute went into hyperparameter optimization for NoRGa w.r.t. Hide? And given the same search budget, how does HiDe-Prompt compare relatively? This is the most crucial aspect, as the other method comparisons are somewhat redundant, since HiDe-Prompt introduces orthogonal classifier head alignment which can be applied to all other methods as well. > It would be great if the authors could address both the placement as a continual learning contribution (as opposed to a parameter-efficient finetuning paper, which would need to be evaluated on other respective benchmark tasks), and the comparability to HiDe-Prompt. __[2]__ This paper also only tackles prompt-based rehearsal-free continual learning, but misses discussions of recent works on first-task-adaptation and simple PEFT-style tuning for Continual Learning, such as: [1] Jason et al. 2022, “A simple baseline that questions the use of pretrained models for continual learning”, [2] Panos et al 2023, “First session adaptation: A strong replay-free baseline for class-incremental learning”, [3] Zhou et al. 2023, “Revisiting class-incremental learning with pretrained models: Generalizability and adaptivity are all you need”, [4] McDonnell et al. 2023: “RanPAC: Random projections and pretrained models for continual learning”. > Given that these works show that existing prompt-style CL objectives are matched by simply training on the first task, and how performance is matchable with simple parameter-efficient finetuning without explicitly accounting for the continual nature of the problem, how should the insights in this paper be rated w.r.t. These works? __[3]__ A vast part of this paper deals with the sample in-efficiency of standard prefix tuning mechanisms. However, the authors do not provide any efficiency tests (e.g. how does performance vary as a function of training iterations / samples seen), and only provide slightly improved performances on a few smaller-scale benchmarks. This is somewhat unfortunate, as it would be great to see how the efficiency estimates (e.g. Eq. 16) connect to practical convergence behaviour (e.g. L288 “polynomial number of data versus exponential number of data”). __[4]__ Both statistical efficiency proofs also do not take into account the continual nature of the problem, but rather assume a single generative model and i.i.d. Samples from said model, which are presented to the learner. This fully disconnects from the actual continual nature and sequential distribution shifts expected in such problems. Similarly, I’m having general troubles to connect the assumptions made throughout the proofs to the practical nature of the problem. E.g. why is Eq. 13 a reasonable assumption for the CL problem? Is convergence behaviour of a LS regression estimator suitably connected to practical convergence behaviour? If so, why? If not, why not? Similarly, why would I want to encourage algebraic independence - what would this give within the actual continual learning scenario? --- __Smaller issues:__ * I find it somewhat difficult to follow the proofs in 4.2 and Appendix A. Providing a proof-sketch as an overview would significantly help readability. * It is a bit problematic to have the entire proof of convergence, without even a proof sketch, in the supplementary- as it motivates the entire section 4 and the non-linear residual gating. It would be great if the authors could provide at the very least a proof sketch in the main paper. * L139/140 is fairly hard to parse.

Questions

All relevant questions are incorporated in the previous section. I am currently leaning towards rejection - not because of the proposed method and the theoretical support (both of which I believe are sensible and meaningful), but rather its placement and consequent evaluation as a continual learning method. Neither in the methodological nature NOR the actually conducted proofs do the author account for the continual nature of the problem. Instead, I believe that this paper is much better suited as a contribution in the domain of parameter-efficient model finetuning. However, this in turn requires different benchmark evaluations. Moreover, I am uncertain about the significance of the report results (see above). Together, I would be happy to raise my score if the authors can address these particular concerns (as well as those listed above).

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors do not discuss limitations or societal impact explicitly.

Summary

The topic of this paper is about the prompt-based continual learning. The authors give a theoretical analysis on these prompt-based continual learning methods, and utilize a Mixture-of-Expert (MoE) architecture characterized by linear experts and quadratic gating score functions. They develop a gating mechanism Non-linear Residual Gates (NoRGa) for MoE-based continual learning. The proposed method has been evaluated on several benchmarks.

Strengths

+ The paper is well-written and easy to follow. + It is interesting to theoretically analyze the effectiveness of prompt-based continual learning. + Continual learning via MoE architecture is worth exploring.

Weaknesses

- Comparison to different Parameter-Effiecient-FineTuning methods [b] (e.g. adapter) is needed. This paper mainly focuses on the theoretical analysis of prompt-based continual learning methods. Prompt-based continual learning belongs to PEFT methods and also add new parameters for new tasks. Can the authors further analyze the advantages of prompt-based methods theroretically? - The parameters cost is usually considered in practical memory-constrained continual learning scenarios. Dynamic routing mechanism can be employed for gating-based neural networks (e.g. [c]). To improve the parameter efficiency of the final model, how to integrate this mechanism in the proposed method? - Important MoE-related continual learning methods are not included in the related works [a]. The difference between the proposed method and other related works should be highlighted. - Performance improvement on some datasets is limited. For example, FA metric (75.06%->75.40%) on Split ImageNet-R, CA metric (95.02%->95.11%) on Split-CIFAR-100 in Table 1. What are the running times of each experiment? Can the authors provide the performance variance of experimental results (e.g. the accuracy at the last incremental learning session)? [a] Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters, CVPR2024 [b] Continual Learning with Pre-Trained Models: A Survey, IJCAI2024 [c] Harder Tasks Need More Experts: Dynamic Routing in MoE Models, arxiv2024

Questions

My major concerns are included in the above weaknesses.

Rating

6

Confidence

5

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors have addressed the limitations.

Re: Rebuttal

I thank the authors for their detailed feedback, which has helped clarified some confusion I had - particularly regarding the placement of the proposed approach in the continual learning literature. Some questions however still remain: * If I understand the reply correctly, this paper directly builds on top of HiDE-Prompt; but does not modify any of the CL components (referring to elements that explicitly account for the continual nature; i.e. the task-specific prompts); but with an explicit focus on improving the WTP (within-task performance). But how is what the authors propose then not just simple parameter-efficient finetuning - simply applied to simplistic CL benchmarks on top of HiDE-Prompt? * Consequently, following Zhou et al. 2023 [3]: What would happen if other PEFT approaches are used to optimize the WTP performance? Or more generally, what does the comparison to first task adaptation sanity checks look like (which are void of meaningful CL elements)? Or a comparison directly to e.g. Zhou et al? Just stating that _"However, these methods, which finetune only the backbone for the initial task, may not always ensure satisfactory separation of new tasks' features."_ may not be sufficient.

Official Comment by Reviewer MKaE

Thank you for the author's positive response. Most of my doubts have been resolved, but in fact I have the same doubts as Reviewer dPHR about "the placement of this work as a Continual Learning contribution". I hope the author will seriously consider whether the proposed method can truly solve the core problems such as catastrophic forgetting from the perspective of continuous learning. In addition, the experimental settings are also compared with some popular PTM-based continuous learning methods. Nevertheless, I also appreciate that this work rethinks the relationship between MoE and prompt from a novel and meaningful perspective, so I still maintain the original score.

Rebuttal Response

Thank the authors for their response. Since the authors have addressed all of my concerns, I decided to increase my score.