FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations

Looking forward to your response

Dear Reviewer, We wanted to inform you that we have addressed the additional concerns you raised. As the rebuttal phase is coming to an end, we kindly request your prompt feedback on our rebuttal. We appreciate your time and consideration and look forward to your response. Best regards, Authors

Thanks for your reply, we clarify the computation and communication cost here

Compared to storing the LoRA modules in a set, stack-based methods can significantly reduce computational costs due to the following advantages: 1. **Reduced Memory Overhead**: During the weight recovery process of LoRA, each LoRA module pair is expanded to the full parameter size. In LoRA fine-tuning with \( M \) clients, if the modules are stored in a set, each module must be recovered separately, leading to a total memory usage of \( (M + 1)P_{\text{full}} \), where \( P_{\text{full}} \) represents the full parameter size. This includes the original parameters and \( M \) newly recovered parameters. In contrast, the stack-based method employed in FLoRA allows for the recovery of the full parameters with **only a single matrix multiplication**, resulting in a total memory requirement of just \( 2P_{\text{full}} \). Given that \( M > 1 \) in federated fine-tuning, this approach leads to substantial computational savings. 2. **Optimized GPU Utilization**: Since parallel computing on a GPU is significantly faster than sequential processing, it is more efficient to compute the stacked LoRA modules collectively rather than individually. This approach better aligns with the GPU’s architecture, leading to faster computations and more efficient resource utilization. To illustrate the benefits of our method in terms of computation and communication costs, particularly in comparison to the baseline FedIT and the set-based method proposed by the reviewer, we provide a table below. For reference, the communication and computation costs of FLoRA are normalized to 1: ========================================================================================= Method $\quad$ Communication $\quad$ Recovering Computation $\quad$ Accurate aggregation $\quad$ Applicable to inference ------------------------------------------------------- FedIT $\quad$ $\quad$ P_{full}/P_{LoRA} $\qquad$ $\qquad$ 1 $\qquad$ $\qquad$ $\quad$ $\qquad$ $\qquad$ No $\quad$ $\qquad$ $\qquad$ $\qquad$ $\qquad$ Yes Set-saving $\qquad$ $\quad$ 1 $\qquad$ $\qquad$ $\qquad$ $(M+1)/2$ $\ $ $\ $ $\ $ $\qquad$ $\quad$ $\qquad$ Yes $\quad$ $\qquad$ $\qquad$ $\qquad$ $\qquad$ No FLoRA (ours) $\quad$ $\quad$ 1 $\qquad$ $\qquad$ $\qquad$ $\quad$ 1 $\quad$ $\quad$ $\qquad$ $\qquad$ $\quad$ $\qquad$ Yes $\quad$ $\qquad$ $\qquad$ $\qquad$ $\qquad$Yes ========================================================================================= According to this table, our approach generally outperforms the baselines in important metrics.

Looking forward to your further feedback

Dear Reviewer 4FRM, The rebuttal deadline is approaching soon, and we have provided detailed responses to your latest concerns. We kindly request that you review our replies at your earliest convenience. If there are any further questions or issues, please let us know. If our responses have adequately addressed your concerns, we would appreciate a higher score. Thank you for your time and consideration.

Thanks for your response. We have addressed your concerns here.

>I am confused about the necessity of adopting a stack-based data structure to merge the local A_i and B_i modules. It seems that we can just adopt a set to store all modules. While storing all modules in a set does indeed have the same memory cost as our proposed method, it remains impractical in real-world scenarios for two primary reasons: **(1) Incompatibility with LoRA Implementation:** LoRA is not implementable when stored separately. In LoRA-fine-tuned language models, the LoRA modules can be utilized without merging them into the base model parameters, allowing the LoRA modules and the original model parameters to be stored together. As discussed in the LoRA paper [1], the forward pass is represented as: $h = \Delta x + W x=W x+BAx,$ where $BA$ represents the LoRA parameter. In our FLoRA approach, the A and B are global LoRA modules. However, if we store all modules separately, the inference stage would require the computation: $h = \Delta x + Wx = Wx + B_0A_0x + B_1A_1x + B_2A_2x + … + B_{K-1}A_{K-1} x,$ This necessitates applying $K$ LoRA modules within a single base model, which introduces iterative computations that are inefficient and impractical during the inference stage. Moreover, current LoRA codebases do not support this approach, where a single base model integrates multiple LoRA modules. The inconvenience of using a set to store LoRA modules becomes particularly problematic as the number of clients increases. **(2) Incompatibility with Multi-Round Federated Fine-Tuning:** Storing LoRA modules to a set cannot effectively support multi-round federated fine-tuning. In subsequent rounds, fine-tuning requires an updated base model, necessitating updating the base model on the server or the clients. Updating on the server, as discussed in our rebuttal, incurs significantly higher communication costs. Conversely, updating on the clients imposes a substantial computational burden on each client, making this approach impractical. >It appears that the communication cost is linear to the number of selected clients and the rank of A and B modules. Could the method still save communication costs when the number of selected clients is large or the rank is large? Yes, we are indeed able to save communication costs when using a widely applied LoRA rank. LoRA is commonly employed for fine-tuning models because it significantly reduces computational resource demands. This advantage hinges on the premise that the LoRA rank is much smaller than the rank of the model parameters themselves. The key prerequisite for using LoRA is that the number of parameters in LoRA must be substantially smaller than that of the base model. Otherwise, the benefits of LoRA would be negated, making full fine-tuning a more efficient option. Therefore, the LoRA rank should not be so large that it surpasses the communication resources required for transmitting the model’s original parameters. Regarding the number of selected clients, it is important to note that the commonly used LoRA rank is typically about 1/100 of the base model size. For instance, in the case of Llama-7b, which has 4096x4096 full parameters, a typical LoRA configuration might use only 16x4096 parameters. This relationship can be expressed as: $P_{LoRA} << P_{full}$ ,where $P$ is the parameter size. When the clients send parameters to the server, FLoRA can consistently save communication costs because: $P_{LoRA} < P_{full}$ This inequality holds true whenever LoRA is used. Moreover, when the server sends parameters back to the clients, FLoRA can also save communication costs under the condition: $2K < P_{full}/P_{LoRA}$ Given that $P_{full}/P_{LoRA}$ is typically greater than 100 in fine-tuning scenarios, and federated fine-tuning server generally lack the communication resources to support extensive client communication (e.g., current works use 10 clients [2]), FLoRA proves effective in reducing communication cost. References: [1] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., ... & Chen, W. (2021). Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. [2] Chen, J., Xu, W., Guo, S., Wang, J., Zhang, J., & Wang, H. (2022). Fedtune: A deep dive into efficient federated fine-tuning with pre-trained transformers. arXiv preprint arXiv:2211.08025.