Library

AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Experts

We introduce AdaMoLE, a novel method for fine-tuning large language models (LLMs) through an Adaptive Mixture of Low-Rank Adaptation (LoRA) Experts. Moving beyond conventional methods that employ a static top-k strategy for activating experts, AdaMoLE dynamically adjusts the activation threshold using a dedicated threshold network, adaptively responding to the varying complexities of different tasks. By replacing a single LoRA in a layer with multiple LoRA experts and integrating a gating function with the threshold mechanism, AdaMoLE effectively selects and activates the most appropriate experts based on the input context. Our extensive evaluations across a variety of commonsense reasoning and natural language processing tasks show that AdaMoLE exceeds baseline performance. This enhancement highlights the advantages of AdaMoLE's adaptive selection of LoRA experts, improving model effectiveness without a corresponding increase in the expert count. The experimental validation not only confirms AdaMoLE as a robust approach for enhancing LLMs but also suggests valuable directions for future research in adaptive expert selection mechanisms, potentially broadening the scope for optimizing model performance across diverse language processing tasks.

cs.CLTopic ModelingNatural Language Processing TechniquesMachine Learning in HealthcareComputer Science
View graphSource

Paper

Similar papers

TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language ModelsLin Mu, Haiyang Wang, Li Ni et al.2026 In Library
MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient Fine-Tuning of Large Language ModelsJie Cao, Tianwei Lin, Hongyang He et al.2025 · Annual Meeting of the Association for Computational Linguistics · 2 citations In Library
DynMoLE: Boosting Mixture of LoRA Experts Fine-Tuning with a Hybrid Routing MechanismD. Li, Naizheng Wang, Zihao Zhang et al.2025 · arXiv (Cornell University) · 9 citations In Library
DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts ModelsGuanzhi Deng, Bo Li, Ronghao Chen et al.2026 · 2 citations In Library
AlphaLoRA: Assigning LoRA Experts Based on Layer Training QualityPeijun Qing, Chongyang Gao, Yefan Zhou et al.2024 · Conference on Empirical Methods in Natural Language Processing · 28 citations In Library
TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language ModelsLin Mu, Haiyang Wang, Li Ni et al.2026 · Annual Meeting of the Association for Computational Linguistics · 1 citation In Library
MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task LearningDacao Zhang, Kun Zhang, Shimao Chu et al.2025 In Library
S'MoRE: Structural Mixture of Residual Experts for Parameter-Efficient LLM Fine-tuningHanqing Zeng, Yinglong Xia, Zhuokai Zhao et al.2025 · Neural Information Processing Systems · 2 citations In Library
MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task LearningDacao Zhang, Kun Zhang, Chu, Shimao et al.2025 · arXiv (Cornell University) · 12 citations In Library
MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuningJ. Zhang, Yiyi Zhao, Dan Chen et al.2024 In Library

Peer review

Acceptcolmweb.org 2024 · via OpenReview
Reviewer yPvd6/10 · confidence 4/52024-04-28

Summary

The paper introduces AdaMoLE, a method using an Adaptive Mixture of Low-Rank Adaptation (LoRA) Experts. Unlike traditional methods that use a static top-k strategy for activating experts, AdaMoLE incorporates a dynamic approach by utilizing a threshold network that adjusts the activation threshold based on tasks. This method employs a gating function combined with the adaptive threshold mechanism to select and activate the most suitable experts according to the input context.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

1. The empirical results showing that AdaMoLE surpasses baseline performances on multiple benchmarks indicate substantial gains in effectiveness, primarily attributed to the adaptive selection of experts tailored to specific contextual needs.

Reasons to reject

1. The experiments are exclusively conducted using a single type of foundation model, LLama2, which may limit the generalizability of the findings. Expanding the range of models tested could provide a more comprehensive understanding of how the proposed methods perform across different foundational architectures. 2. The proposed method's approach to selecting a hyperparameter using a network-provided threshold has previously been explored. Furthermore, given that using a threshold for expert selection is already established in the literature, the introduction of a network to determine this threshold does not substantially advance the existing methodologies.

Questions to authors

It would be beneficial if the authors could extend their experiments to include additional foundation models, such as Gemma.

Reviewer eBvp6/10 · confidence 4/52024-05-11

Summary

This paper proposes a novel adaptive mixture of experts LoRa tuning solution. The key contribution lies in the introduction of a learnable gate function and a threshold mechanism. Specifically, the authors develop a novel threshold function, which consists of a single linear layer followed by a sigmoid function, and it is modulated by a hyperparameter scalar, denoted as $\tau_{max}$. The experiments, conducted using the LLAMA 7B model with a LoRa rank set to 32, demonstrate that this adaptive mixture of experts LoRa sometimes outperforms the established baselines.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

The proposed solution is both intuitive and innovative. The results analysis offers some insightful findings, enhancing the value of the study. Additionally, the writing is clear and easy to follow.

Reasons to reject

The experiments presented in this paper require further improvement to enhance the soundness of the results: Lack of Relevant Baselines: The paper does not include comparisons against key LoRa baselines and mixture of experts LoRa baselines, such as SiRA [1]. The absence of these baselines prevents a thorough quantification of the contributions made by this study. Limited Experimental Settings: Both LoRa and the mixture of expert LoRa are sensitive to the rank and number of experts. The authors should explore different hyperparameter settings for LoRa and mixture of expert LoRa to fully understand their impacts. Additionally, the current results suggest that performance is particularly sensitive to the hyperparameter $\tau_{max}$ . It is recommended that the authors test a wider range of settings and provide more comprehensive findings and insights to strengthen the validity of their results. Lastly, it is essential to conduct experiments using different pretrained models. Testing the proposed solution across a variety of models will help validate its effectiveness and generalizability in diverse settings. [1] SiRA: Sparse Mixture of Low Rank Adaptation

Reviewer pQdU6/10 · confidence 4/52024-05-11

Summary

Overview: This paper introduces a mixture-of-experts style of low-rank adaptation fine-tuning of large language models where the learned threshold function adjusts the number of activated experts LoRA weights dynamically. The synergy of PEFT techniques and MOE architectures brings out the moderate improvement of a 7b LM on commonsense reasoning and language understanding datasets. Overall speaking, this paper is clearly written, the idea is quite straight-forward and intuitive while the experiment scope is kinda limited.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

1. Novel dynamical routing of LoRA experts when dealing with each input token by means of a learned threshold function.

Reasons to reject

1. Incomplete evaluation: Several crucial experimental settings are missing: a) Only commonsense reasoning and basic language understanding tasks are evaluated. Coding, mathematic reasoning and advanced LU and QA tasks (eg. MMLU and GPQA) are not involved. b) Only one moderate-sized model (llama2-7b) is tested, which brings about the doubt whether the proposed method can be well suitable for much larger language models. c) Only one rank (ie. 32 or 4 * 8) setting is discussed. d) \``[0, 1/(2N)]" is the best threshold interval for the proposed method. From Fig. 3, the number of actual activated experts is quite large (almost >=6 for all layers). It seems that the computational efficiency and GPU memory occupation of AdaMoLE are flawed compared to typical top-2 routing mechanism. Even the interval of \``[0, 1/N]" has the same problem in the shallow transformer layer.

Questions to authors

Q: I am worried about the rank deficiency of MOE of LoRA matrices since the rank of sum of 8 expert matrices with rank 4 will be smaller than a single LoRA expert matrix with rank 32. If the rank collapses, the adaptability power of MOE-LoRA fine-tuning will be restricted. So is there any academic materials or instructional insights covering this?

Reviewer eBvp2024-06-05

I would like to **increase the score from 5 to 6** due to the additional experiment results presented in the rebuttal. However, these experiments are currently not extensive. I suggest the authors further improve the experiments for the next version.

Authorsrebuttal2024-06-05

Thank you for your feedback and for raising our score. We appreciate your suggestion to enhance our experiments and will work on incorporating more extensive evaluations in the next version.

Reviewer yPvd2024-06-05

The authors' responses addressed my concerns related to foundational models. Hence, I increased my score.

Authorsrebuttal2024-06-06

We appreciate your concern regarding the rank deficiency of MoE-LoRA matrices. To clarify, let's denote $N$ as the number of experts and $r$ as the rank of each LoRA matrix $A_i$ and $B_i$. For each input $x$, the combined MoE-LoRA matrix is given by $\sum_{i=1}^N w_i(x) B_i A_i$. While each $B_i A_i$ has a rank $r$, the combined rank of their weighted sum can be up to $N \times r$ if they span independent subspaces, as $\text{rank}\left(\sum_{i=1}^N w_i(x) B_i A_i\right) \leq \sum_{i=1}^N \text{rank}(B_i A_i) = N \times r$. In the case of top-k gating (or threshold gating), only the top-k experts are selected for each input, resulting in a rank of at most $k \times r$. Although $k \times r$ is smaller than $N \times r$, empirical results show that MoE-LoRA models still achieve high performance. This is because the gating mechanism dynamically selects the most relevant experts for each input, where each small matrix $B_i A_i$ specializes in different features. The dynamic selection mechanism ensures that the most relevant experts are activated based on the input $x$, allowing the model to effectively use the most informative subspaces tailored to the specific input. Consequently, even with a lower theoretical rank, the adaptive combination of small LoRA matrices can still outperform a single large LoRA matrix. We hope this explanation clarifies our approach. We appreciate your review and look forward to any further questions or comments you may have.

Reviewer pQdU2024-06-06

Appreciate the authors' additional experiments w.r.t. the evaluation issues. Adaptive selection to adjust the lora experts' parameters is a synergy of already well-known approaches. I will maintain my rating since the novelty lies in the combination rather than creation of methods. However, I would like to respect the opinions of other reviewers and chairs.

Reviewer pQdU2024-06-06

In the above authors' hyper-parameter test, AdaMoLE seems to saturate with the increase of lora expert number or rank, while other methods tend to have a higher performance with the total theoretical rank (i.e. $N\times{r}$). However, the overall competitiveness of proposed method is quite obvious. Thanks for the authors' response to my rank-deficiency concern of extending lora fine-tuning effectiveness by moe architecture. However, the response is almost an empirical standpoint which may be influenced by quite a lot experimental setting issues. And to clarify my concern, the rank-deficiency event may at most take place 7 times when assembling 8 rank-4 lora matrices (which I would like to name as ``low-rank ensemble issue"). Maybe the rank restoring technique proposed by ReLoRA[1] can be injected in place to MoE-LoRA, e.g. refreshing lora experts' weights according to a schedule. [1] ReLoRA: High-Rank Training Through Low-Rank Updates: https://arxiv.org/abs/2307.05695

Authorsrebuttal2024-06-07

Thank you for your detailed feedback and for acknowledging our additional experiments. We appreciate your insights on the synergy of existing methods and the constructive suggestion regarding the rank-restoring technique from ReLoRA. We will consider these perspectives for future work. Thank you again for your review and thoughtful comments.

Program Chairsdecision2024-07-10

Decision

Accept

© 2026 NYSGPT2525 LLC