Differentially Private Zeroth-Order Methods for Scalable Large Language Model Finetuning

Fine-tuning large language models (LLMs) on downstream tasks has become a standard approach to adapt their capabilities. However, the process raises privacy concerns when using sensitive datasets, prompting increasing interest in differentially private (DP) fine-tuning methods. While existing approaches build upon the seminal work of DP-SGD, they are constrained by the inherent inefficiency bottlenecks. In this paper, we investigate the potential of DP zeroth-order methods for LLM fine-tuning, which avoids the scalability bottleneck of SGD by approximating gradients with more efficient zeroth-order gradients. We propose the stagewise DP zeroth-order method (DP-ZOSO) that dynamically schedules key hyperparameters to leverage the synergy between DP random perturbation and the gradient approximation error. To further enhance the scalability, we propose DP zeroth-order stagewise pruning method (DP-ZOPO) which reduces the trainable parameters by a data-free pruning technique requiring no extra privacy budget. We provide theoretical analysis for both proposed methods and conduct extensive empirical analysis on both encoder-only masked and decoder-only autoregressive language model, achieving impressive results in terms of scalability and utility across diverse tasks.