FL-DPLoRA: An Integrated and Efficient Privacy-Preserving Training Framework for Large Language Models in Privacy-Critical Applications

Abstract

In the era of large language model (LLM) applications, the widespread deployment of intelligent systems in privacy-critical domains (e.g., mental health, finance) poses significant challenges for data privacy and the limited availability of labeled data. To address these issues, FL-DPLoRA integrates federated learning (FL), transfer learning (TL), and differential privacy (DP) into a unified training paradigm. Specifically, it leverages a public dataset to pretrain a base model and then employs a low-rank adaptation (LoRA) mechanism for local fine-tuning at each client, with Gaussian noise injection during gradient aggregation to enforce rigorous differential privacy guarantees during updates. Our framework significantly reduces communication overhead by transmitting only a small set of adapted parameters while preserving data confidentiality and minimizing communication costs. We provide formal privacy guarantees and demonstrate the effectiveness of FL-DPLoRA through comprehensive experiments in mental health detection tasks. The results show that FL-DPLoRA achieves competitive performance with minimal degradation under strict privacy budgets and reduces communication costs by more than 99.7% compared to conventional methods. This validates FL-DPLoRA as a broadly applicable solution for safely deploying LLMs in sensitive, distributed environments.