Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuning

Summary

The paper delves into the issue of privacy concerns when fine-tuning large language models (LLMs) on sensitive data. The paper introduces the concept of User-level Differential Privacy (DP) as a solution to ensure uniform privacy protection across users, especially in scenarios where contributions per user vary. By focusing on two mechanisms, Group Privacy and User-wise DP-SGD, the study evaluates the design choices such as data selection strategies and parameter tuning to strike a balance between privacy and utility. Overall, the paper provides a new perspect of user-wise DP for LLM fine-tuning on natural language generation tasks, shedding light on the importance of considering user privacy guarantees in addition to example-level privacy. However, the solution is somehow lacking of novelty. Moreover, the comparison of proposed approach is not thorough.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

1). The work is easy to follow and well-organized: The paper on User-Level Differential Privacy for Language Model Fine-Tuning excels in its clarity and organization, making it easy for readers to follow the research methodology, findings, and implications. The paper is structured in a logical manner, with clear section headings and transitions that guide the reader through the exploration of User-level DP mechanisms. By presenting a systematic evaluation of privacy protection strategies in a coherent and accessible manner, the paper ensures that both experts and non-experts in the field can grasp the significance of the research . 2). The problem of privacy protection is an important issue: The paper underscores the critical importance of privacy protection in the context of fine-tuning large language models on sensitive data. With the proliferation of advanced machine learning models and the potential risks of memorization when handling personal or confidential information, ensuring robust privacy guarantees has become a paramount concern in the field of artificial intelligence and data science.

Reasons to reject

1). The solution is somehow straightforward building upon the existing groupwise DPSGD which limits the contribution of the submission. 2). the comparison of proposed approach is not thorough. Many other DP approaches are not compared such as: LARGE LANGUAGE MODELS CAN BE STRONG DIFFERENTIALLY PRIVATE LEARNERS https://openreview.net/pdf?id=bVuP3ltATMz To name a few. 3). Typos: Page 3: a Example-level==>an Example-level 4). The method needs the present/availablility of user info, which might be missing in many real applications.

Questions to authors

1). More thorugh comparisons to other SOTAs of DP. 2). What if the user info is not available for different samples?

Summary

In this paper, the authors present a study on the application of user-level differential privacy (DP) in the context of fine-tuning large language models (LLMs) on natural language generation tasks. It is a critical issue in the field of machine learning, specifically the privacy concerns associated with fine-tuning LLMs on sensitive data. The document is well-structured, with a clear abstract, introduction, methodology, experimental setup, results, and conclusions. Experimental results on Enron Email dataset and BookSum dataset were given to show the effectiveness of the proposed method.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

1. The paper tackles the concept of user-level DP in the fine-tuning of LLMs, which is a departure from the traditional example-level DP. 2. The authors provide a systematic empirical evaluation of two user-level DP mechanisms.

Reasons to reject

1. The paper mentions that User-wise DP-SGD may require modifications to the data loading pipeline, which could be a barrier to adoption in practice. 2. The computational overhead introduced by the differential privacy mechanisms is mentioned but not thoroughly analyzed, particularly for larger models or datasets.

Summary

In this work, the authors study language model fine-tuning with differential privacy guarantees. Since each user generally contributes multiple text sequences to the training corpus, to ensure equal privacy guarantees across users, the privacy unit needs to be set at the user level. The authors identify and systematically evaluate two mechanisms, Group Privacy and User-wise DP-SGD, for achieving user-level differential privacy when fine-tuning large language models on natural language generation tasks. For each mechanism, the authors investigate data selection strategies to improve the trade-off between privacy and utility.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

1. The authors present a systematic empirical evaluation of User-level DP for fine-tuning LLMs on natural language generation tasks. 2. The authors therefore study user-level DP motivated by applications where it necessary to ensure uniform privacy protection across users.

Reasons to reject

1. The technical novelty is limited. What is the challenge to adapt User-wise DP-SGD in LLM, there are many existing User-wise DP-SGD methods. 2. Lack of baselines. It is better to add some recent baseline to demonstrate the advantage of the proposed method. For example: Large Language Models Can Be Strong Differentially Private Learners. 2022 ICLR

Questions to authors

see weakness.

Follow-up

Dear Reviewer sVEv, Thank you once again for taking the time to share your suggestions on our work. As the discussion period is nearing its end, we want to ensure that we have adequately addressed all your comments. Specifically, do you still have concerns regarding: - Q1: Our contribution, where we contribute a **comprehensive evaluation suite of user-level DP implementations for LLMs and well-motivated data selection strategies**. - Q2: Performance when integrated with other DP fine-tuning methods, where we present **new results for our evaluation when incorporated into the DP fine-tuning pipeline you suggested**. - Q3: The availability of user information in real applications, where we adhere to the **standard user-level DP notion** that can still function as long as there is a mechanism to group examples based on user ownership. We look forward to your feedback. Best regards, Authors of Submission933

Follow-up

Dear Reviewer pxQP, Thank you once again for taking the time to share your suggestions on our work. As the discussion period is nearing its end, we want to ensure that we have adequately addressed all your comments. Specifically, do you still have concerns regarding: - Q1: The implementation of user-wise DP-SGD, where we clarify that it **can be implemented with minimal effort**. - Q2: The computational overhead, where we report the training time for your reference and observe **minimal computational overhead** (4%–9%) compared to non-private training for the GPT-2 model of various sizes. We look forward to your feedback. Best regards, Authors of Submission933