DP-DyLoRA: Fine-Tuning Transformer-Based Models On-Device under Differentially Private Federated Learning using Dynamic Low-Rank Adaptation

Federated learning (FL) allows clients in a communication network to collaboratively train a global model without sharing their local data with other parties. Nevertheless, clients’ contributions to the server can still leak sensitive information about the underlying data. To address this, differential privacy (DP) provides formal privacy guarantees by introducing random noise into clients’ contributions. However, the added noise makes it challenging to train models effectively. Recent studies have therefore attempted to alleviate this issue by leveraging parameter-efficient fine-tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA), which reduce the dimensionality of clients’ contributions and thereby mitigate the impact of the added noise. Despite these efforts, significant performance degradation remains a major concern. In this paper, we propose differentially private dynamic low-rank adaptation (DP-DyLoRA) to further alleviate the impact of DP noise on training effectiveness and reduce the communication overhead by dynamically updating a subset of LoRA parameters. We conduct a comprehensive evaluation of both baseline and state-of-the-art (SOTA) PEFT methods in DP-FL settings. Experimental results show that DP-DyLoRA outperforms SOTA methods by an average of 12.1% in test accuracy for a stringent privacy budget of ϵ = 2.