Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternatives
While open Large Language Models (LLMs) have made significant progress, they still fall short of matching the performance of their closed, proprietary counterparts, making the latter attractive even for the use on highly private data. Recently, various new methods have been proposed to adapt closed LLMs to private data without leaking private information to third parties and/or the LLM provider. In this work, we analyze the privacy protection and performance of the four most recent methods for private adaptation of closed LLMs. By examining their threat models and thoroughly comparing their performance under different privacy levels according to differential privacy (DP), various LLM architectures, and multiple datasets for classification and generation tasks, we find that: (1) all the methods leak query data, i.e., the (potentially sensitive) user data that is queried at inference time, to the LLM provider, (2) three out of four methods also leak large fractions of private training data to the LLM provider while the method that protects private data requires a local open LLM, (3) all the methods exhibit lower performance compared to three private gradient-based adaptation methods for local open LLMs, and (4) the private adaptation methods for closed LLMs incur higher monetary training and query costs than running the alternative methods on local open LLMs. This yields the conclusion that, to achieve truly privacy-preserving LLM adaptations that yield high performance and more privacy at lower costs, taking into account current methods and models, one should use open LLMs.
Paper
Similar papers
Peer review
Summary
The paper critiques the practices of privately adapting closed(-source) LLMs to private data by demonstrating that these techniques are potentially unsafe and do not yield the required quality of resulting model in terms of accuracy. The authors conclude that a focus on open LLMs should be preferable in sensitive fields.
Strengths
I agree with both the motivations and the premise of this paper. It is important to think about adapting LLMs to private data holistically, i.e. on a systems level, and not just on the level of individual techniques. I like that the authors investigate a multitude of techniques, contrast open and closed LLMs, and discuss the costs of each method. The paper is also written clearly and includes a good overview of prior works. I also approve of the detailed exposition on hyperparameters, costs, etc. in the appendix.
Weaknesses
I would have liked to see a more detailed investigation on the effect of privacy levels beyond $\varepsilon=8$ (for most techniques excluding PATE, where there is a note on different privacy levels and the performance plateauing). Perhaps I missed this, but I didn't find a concrete justification for $\varepsilon=8$ either, although I recognise this as a "folklore/default" privacy budget which many of us have come to think of "automatically" when working with DP-SGD due to the multitude of works which use it. Moreover, it would have been nice to see more dedicated instruction fine-tuning tasks, since this arguably has become one of the most important tasks for contemporary LLMs. On a more conceptual level: There is no "methodological or theoretical novelty" in this work in the traditional sense: It is a (thorough and thoughtful) comparison of techniques. I don't personally view this as a big weakness, to the contrary, I disapprove of the, usually fairly arbitrary, notions of "novelty" used to argue against papers, and I would like to see this paper published in some form, as I believe it discusses important points. However, I am unsure whether it would not have been a better fit to the benchmark track, rather than the main conference track.
Questions
- What are the "privacy units" in this work? In other words, how are adjacent databases defined, and how is this selection justified? - Privacy agains the LLM provider is a strong argument. I'd be interested in some more discussion on the threat model also in terms of releasing a privately fine-tuned model and the types of attacks which are possible, e.g. MIA or reconstruction attacks, and how they would be carried out. To clarify: I am not asking you to perform any attacks. - As discussed above, your privacy budget evaluation tops out at 8. I would like to see some discussion on protecting e.g. against reconstruction attacks, which is feasible even with much higher privacy budgets. This should be part of the discussion on potential attacks, and I'd be interested to see when the performance between models converges to each-other and to the non-private performance - There don't seem to be mixture of experts models involved in your evaluation. Do you believe that this is a limitation of your work? Would you have anticipated similar results and why? I'd welcome a discussion on this point. - What interactions between privacy and safety did you observe? Do you anticipate that solving a privacy problem could introduce an alignment or safety problem in these models?
Rating
7
Confidence
4
Soundness
3
Presentation
4
Contribution
3
Limitations
The authors discuss some limitations, such as cost and performance, which I appreciate. I would be interested (see Questions above) in a discussion on the interplay between privacy and other aspects of trustworthiness, especially safety and alignment.
Thank you for the prompt response & such positive feedback!
We thank the Reviewer for the prompt response and are pleased to receive such positive feedback. We also appreciate that the Reviewer supports our work and increased the score. >**On the topic of privacy budgets beyond $\varepsilon=8$: I meant $\varepsilon>8$. In particular, since reconstruction attacks can be defended against with very large privacy budgets ($O(1000)$ and above), it would have been interesting to see some results in this direction too. However, as I recognise that I should have written "larger than" and better specified my request, I will not hold this against you, as I also recognise that this is a topic which is tangental to your work.** Thank you for the clarification. The criticism of the previous work was primarily targeting too high $\varepsilon$ values, which led us to concentrate on the $\varepsilon<8$ instead of considering the full rage of the possible $\varepsilon$ values. We are willing to provide the results on reconstruction attacks for larger privacy budgets. Could the Reviewer please point us out to any previous work that we should/could leverage here? Specifically regarding that “reconstruction attacks can be defended against with very large privacy budgets ($O(1000)$ and above)”. Would the Reviewer recommend using a specific large value, such as $\varepsilon = 1000$, to evaluate how adapted LLMs perform in defending against such attacks? >**On the topic of privacy units: Thank you for clarifying this. Please specify it precisely in the final manuscript. A side note in case this may have slipped your attention: The argument you seem to be making about using add one since you are using RDP for accounting does not have a causal basis: RDP can be used with any adjacency notion (e.g. using the Google DP accounting library).** We appreciate this insight and fully agree that RDP can be used with any adjacency notion. Indeed, an alternative definition of adjacency assumes the two neighboring datasets are of equal size and is based on the replacement of a single record in one of the datasets. We should have been more precise in our explanation during the rebuttal. Our evaluation relied on private-transformers which adopted the definition of *neighboring* based on addition/removal [1], which in turn was taken from an earlier work on RDP [2]. We already specified it precisely in our manuscript. **References:** [1] “Large Language Models Can Be Strong Differentially Private Learners”. Xuechen Li, Florian Tramer, Percy Liang, Tatsunori Hashimoto. ICLR 2022 (reference 29 in the main paper). [2] “Rényi Differential Privacy of the Sampled Gaussian Mechanism.” Ilya Mironov, Kunal Talwar, Li Zhang. 2019 (a new reference).
On the topic of large $\varepsilon$ values
> Thank you for the clarification. The criticism of the previous work was primarily targeting too high $\varepsilon$ values, which led us to concentrate on the $\varepsilon<8$ instead of considering the full rage of the possible $\varepsilon$ values. Thank you for clarifying, I understand, and agree that the high privacy regime is very important, I thus see the rationale behind reporting the lower values in your current work. > We are willing to provide the results on reconstruction attacks for larger privacy budgets. Could the Reviewer please point us out to any previous work that we should/could leverage here? Specifically regarding that “reconstruction attacks can be defended against with very large privacy budgets ($O(1000)$ and above)”. Would the Reviewer recommend using a specific large value, such as $\varepsilon = 1000$, to evaluate how adapted LLMs perform in defending against such attacks? Let me preface my response by clarifying that my current score is unconditional of any further experiments you may want to conduct, as I think that doing reconstruction attacks in a rebuttal phase with only a few days left is a big ask. Therefore, please regard my recommendation primarily as a stimulus for future work or an add-on to your paper. In case you would like to try to attempt reconstructions (which would probably be elements of the embedding space, not actual inputs, which adds a layer of complexity here), you could look at the work of Hayes et al., NEURIPS 2023 (https://proceedings.neurips.cc/paper_files/paper/2023/file/f8928b073ccbec15d35f2a9d39430bfd-Paper-Conference.pdf). I would like to stress that the findings from such experiments may not be particularly surprising, in the sense that they will probably yield the (already known fact) that "DP with very large epsilon (I've seen values as astronomical as 10^9 in recent works) protects against input (in your case, probably embedding) reconstruction while not significantly impacting utility".
Reconstruction Attacks
We appreciate the Reviewer’s suggestion, which provides valuable direction for future work. Given that a few private adaptations for LLMs also utilize the DP-SGD algorithm [2] (e.g., PromptDPSGD[5]), further exploration of the recommended paper [1] will undoubtedly be highly interesting. There are also PATE-based [3,4] adaptations, such as PromptPATE [5]. Since these methods rely on public data to train the student prompt, with private data never being used to train any released prompts, reconstruction attacks are likely not feasible. **References:** [1] “Bounding Training Data Reconstruction in DP-SGD.” Jamie Hayes, Saeed Mahloujifar, Borja Balle. NeurIPS 2023. [2] “Deep Learning with Differential Privacy.” Martín Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, Li Zhang. CCS 2016. [3] “Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data.” Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian Goodfellow, Kunal Talwar. ICLR 2017. [4] “Scalable Private Learning with PATE.” Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, Úlfar Erlingsson. ICLR 2018. [5] “Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models.” Haonan Duan, Adam Dziedzic, Nicolas Papernot, Franziska Boenisch. NeurIPS 2023.
Summary
This paper compares the performance and privacy leakage between private adaptations on closed and open-source LLMs. The authors conclude that adaptations on open-source LLMs result in better performance, lower training costs, and enhanced privacy protection
Strengths
The paper presents extensive experiments on adaptations of both closed and open-source LLMs, including four private In-Context Learning (ICL) methods for four closed LLMs and three private tuning methods on four open models.
Weaknesses
1. The work lacks novelty in its methods. All the private adaptation methods used are pre-existing, with the authors only extending two methods from classification tasks to generation tasks. This makes the paper more of a benchmark work rather than original research. Additionally, as noted in section 4.2, "Previous work [31] has shown for non-private settings that gradient-based tuning methods (used for open LLMs) offer better accuracy and significantly lower computational costs than ICL (used for closed LLMs) since the adaptations can leverage the internal behavior of the LLM." Thus, it is intuitive that the performance would be similar under DP scenarios, making the conclusion to use open LLMs less impactful. 2. It seems unnecessary to use private tuning methods on open LLMs since the trained model will not be shared and only queried. As the authors state, "Looking at Figure 1, it becomes obvious that any private tuning method executed on that open LLM would, conceptually, improve privacy protection since the LLM provider would neither be involved in the adaptation nor in the use of the adapted LLM, yielding absolute privacy against them." If a company locally fine-tunes the model and then preserves it locally for queries, direct fine-tuning would be more efficient. 3. The comparison in the experiments is unfair and lacks proper baselines. The experiments compare task performance between various adaptations on closed and open LLMs. However, the closed and open models are different and show varying performance even without fine-tuning. The zero-shot performance of these models is not provided, leaving no baseline for reference. Additionally, DP-based ICL methods and fine-tuning methods offer different levels of privacy protection. As the authors mention, "Yet, the threat model of multiple private ICL methods for closed LLMs does not include providing privacy against the LLM provider." It is not fair to compare their performance with the same parameter $\epsilon$ as shown in Figure 2, which could mislead readers into thinking they offer the same level of privacy protection.
Questions
The authors categorize privacy leakage into Private Training Data and Private Query Data. However, I did not see any data inference in the experiments. Is there any data reconstruction attack or metric used to measure the leakage in your model?
Rating
4
Confidence
4
Soundness
2
Presentation
2
Contribution
1
Limitations
Please refer to the weaknesses and questions sections.
Thank you for the prompt response & positive feedback!
We appreciate the Reviewer’s prompt response and the willingness to help us further improve our submission. We are also happy that we were already able to address most of the Reviewer’s concerns. >**1. I appreciate the use of Differential Privacy (DP) as a framework for measuring privacy and providing guarantees. However, I'm concerned about the comparison setting in your experiments. Given that the closed LLM uses Prompt Tuning (PT) while the open LLM uses model Fine-Tuning (FT), there's insufficient evidence to suggest that applying the same DP parameter $\varepsilon$ to both PT and FT provides equivalent privacy guarantees for the training data. Therefore, using the same $\varepsilon$ across different scenarios may not be appropriate for a fair comparison.** Comparing different methods at the same privacy budget $\varepsilon$ is a standard approach and was used in the previous work. For example, in [1], Table 2 presents the comparison between DP-OPT (which uses hard prompts) and PromptDPSGD (which employs soft-prompt tuning) at the same privacy level $\varepsilon=8$ and the same parameter $\delta=1/|D|$, where $|D|$ denotes the size of the fine-tuning set. Therefore, we believed that this comparison setting was justified. >**2. The use of DP $\varepsilon$ as the sole indicator of privacy leakage level is not fully convincing. As mentioned in the first point, PT and FT may have different privacy risks under the same $\varepsilon$. To provide a more comprehensive evaluation of privacy leakage for both Private Training Data and Private Query Data, I suggest incorporating additional attack methods, such as membership inference attacks or model inversion attacks.** We understand the Reviewer’s concerns. This is also why, in the rebuttal, we referred to previous work [1 (Table 8), 2,3 (Figure 2)] that analyzed the privacy leakage of the LLM adaptations using Membership Inference Attacks (MIAs). To clarify, is the goal to compare the vulnerability of the ICL (In-Context Learning) methods, used to adapt closed LLMs, with the tuning methods (such as fine-tuning, LoRA, or prefix-tuning which rely on access to model gradients), used for open LLMs, in the context of attacks like membership inference? We are willing to run an additional experiment that can address these concerns. One of the candidates is to use the ICL method (i.e., hard prompts) and the tuning method (e.g., LoRA) on the same model (e.g., Vicuna 7B) and then compare the success rate of a Membership Inference Attack (MIA) against these two adaptations. These can be executed for the non-private setting as well as for both methods trained under Differential Privacy (with, e.g., $\varepsilon=8$). If the Reviewer meant another evaluation, we would be grateful for further clarification in the comments below. **References:** [1] DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer Junyuan Hong, Jiachen T. Wang, Chenhui Zhang, Zhangheng Li, Bo Li, Zhangyang Wang. ICLR 2024 (spotlight). [2] “On the privacy risk of in-context learning”. Haonan Duan, Adam Dziedzic, Mohammad Yaghini, Nicolas Papernot, Franziska Boenisch. In The 61st Annual Meeting Of The Association For Computational Linguistics 2023. [3] “Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models”. Haonan Duan, Adam Dziedzic, Nicolas Papernot, Franziska Boenisch. In the Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS) 2023.
Membership Inference Attacks
We ran the additional experiment to address the concerns. We used the ICL method (i.e., hard prompts with PromptPATE) and LoRA on Vicuna 7B. We executed the experiment for the non-private setting ($\varepsilon=\infty$) as well as for both methods trained under Differential Privacy (with, $\varepsilon=8$ for LoRA and $\varepsilon=0.4$ for PromptPATE, which plateaus after this value and the same outcome is obtained when we set $\varepsilon=8$). We followed the experiment from [1] (Figure 2). Our results indicate that the members and non-members are easily distinguishable for non-private adaptations. In contrast, after running the private adaptations, the distribution probabilities of members vs non-members become significantly less distinguishable which makes the membership inference attacks less successful. If the Reviewer meant another evaluation, we would be grateful for further clarification in the comments below and are willing to answer any further questions. **References:** [1] “Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models.” Haonan Duan, Adam Dziedzic, Nicolas Papernot, Franziska Boenisch. NeurIPS 2023.
I have raised the score and still concerned about the novelty, DP comparison, and lack of privacy attack
Thank you for your diligent work and detailed explanation. I appreciate the effort you've put into this paper. After careful consideration, I have a few thoughts I'd like to share: Regarding novelty: While the extension of the two methods to the generation task is interesting, I wonder if we could further explore how this contributes to the field beyond what might be intuitively expected. For instance, the conclusion that fine-tuning outperforms prompt tuning even with DP added seems to align with general expectations. On the membership inference attack: This is a valuable addition to the paper. To strengthen this aspect, have you considered incorporating additional types of attacks? This could provide a more comprehensive view of privacy levels beyond the DP noise. Concerning the DP comparison: The application of DP to both model fine-tuning and prompt tuning is noteworthy. However, Even though previous work compare the soft and hard prompt tuning under same DP noise, it is different from model finetuning and prompt tuning. It is hard to ensure a fair comparison between these two approaches, given their inherent differences. Overall, your paper demonstrates thorough experimentation and is well-written. To further enhance its impact, you might consider addressing the points above, particularly focusing on highlighting the novelty of your approach, expanding the range of privacy attacks examined, and refining the DP comparison methodology. Thank you again for your hard work.
Novelty, Expectations, and Previous Work
>**Thank you for your diligent work and detailed explanation. I appreciate the effort you've put into this paper. After careful consideration, I have a few thoughts I'd like to share:** We greatly appreciate the Reviewer's thoughtful engagement in the discussion period and the careful consideration of our research. >**Regarding novelty: While the extension of the two methods to the generation task is interesting, I wonder if we could further explore how this contributes to the field beyond what might be intuitively expected.** Thank you for recognizing the significance of our two novel methods for the generation tasks. *This was not intuitively expected.* Please check Lines 56 to 60 in our submitted manuscript where we explain: “We demonstrate how to locally apply privacy-preserving prompt-based methods to train generation tasks with high-performance -- **claimed impossible by prior work [ 29 ]**. In particular, we show for the first time that private prompt tuning for text generation tasks can achieve comparable performance to private (full) fine-tuning and private low-rank adaptations (LoRA).” We further write in Lines 138 to 141: “Having prompt-based generation holds the advantage that, in contrast to fine-tuning based approaches, they support mixed-task inference [24, 27, 31], i.e., they require one frozen model for multiple tasks rather than a separate model copy for each of them. This reduces storage and offers greater flexibility and efficiency.” Our main message emphasizes the importance of developing new adaptation methods for open LLMs, particularly focusing on PEFT methods, especially those that are input-based, like prefix tuning. We would appreciate any further suggestions from the reviewer on how we might strengthen the description of how our methods contribute to the field." >**For instance, the conclusion that fine-tuning outperforms prompt tuning even with DP added seems to align with general expectations.** This conclusion is also not aligned with general expectations. The gradient based adaptations (e.g., PromptDPSGD, PrivateLoRA, DP-FineTune) are typically applied to much smaller open LLMs, with models like Llama3-8B containing only up to 8 billion parameters. In contrast, the hard-prompt based methods (e.g., PromptPATE, DP-OPT, DP-FewShotGen) are used with significantly larger LLMs, such as GPT3.5 Davinci with 175B parameters, Claude with 200B parameters (Claude), or even larger models like GPT4-Turbo with 1.76T parameters! The closed models also have much higher zero-shot performance than the open LLMs. However, the adaptation of open LLMs is more powerful, ultimately leading to greater performance gains for specific downstream tasks. Furthermore, [24] claimed the opposite to our results. Specifically, Table 2 in [24] shows that DP-OPT on GPT3.5 Davinci outperforms PromptDPSG [15] on Vicuna 7B. We observe that this is because of the insufficient fine-tuning of the PromptDPSGD method and without further analysis of other private gradient-based adaptation methods. Additionally, we would like to address the previous concerns regarding other related work [31] in more detail below: >**"Previous work [31] has shown for non-private settings that gradient-based tuning methods (used for open LLMs) offer better accuracy and significantly lower computational costs than ICL (used for closed LLMs) since the adaptations can leverage the internal behavior of the LLM." Thus, it is intuitive that the performance would be similar under DP scenarios, making the conclusion to use open LLMs less impactful.** Note that the conclusion (cited above) from [31] is based only on their single Table 1, which includes only a single gradient-based method ($IA^3$) and solely few-shot ICL, both tested exclusively on held-out tasks from T0. For the gradient-based method, they used T0-11B which has better 0-shot performance than GPT3 Da Vinci on the tasks held out from T0. One of the Reviewers of the paper [31] wrote: “The paper achieves strong few-shot results with moderate language model sizes (up to 11B), outperforming more expensive models like GPT3.” However, the fact that $IA^3$ performs better is not surprising. In contrast, **our results are striking**, as highlighted by Reviewer gGr3. We analyzed a broader range of models, datasets, and methods, with a specific focus on privacy-preserving adaptations. Firstly, we show that even larger models like GPT-4 underperform compared to open LLMs like Llama3-8B when adapted, despite the open LLMs starting from a much lower zero-shot performance baseline. Secondly, our results not only consider performance and cost, as in [31], but also emphasize the critical aspect of privacy protection for end-users of LLMs.
Membership Inference Attacks & DP Comparison
>**On the membership inference attack: This is a valuable addition to the paper. To strengthen this aspect, have you considered incorporating additional types of attacks? This could provide a more comprehensive view of privacy levels beyond the DP noise.** Yes, we have considered incorporating additional attacks, including, for example, the exploration of reconstruction attacks. However, as [noted by Reviewer my5W](https://openreview.net/forum?id=Jf40H5pRW0¬eId=oPC5WsBRPB): “I think that doing reconstruction attacks in a rebuttal phase with only a few days left is a big ask.” Given that a few private adaptations for LLMs also utilize the DP-SGD algorithm [2] (e.g., PromptDPSGD[5]), further exploration of the recommended paper on reconstruction attacks [1] will undoubtedly be highly interesting. There are also PATE-based [3,4] adaptations, such as PromptPATE [15]. Since these methods rely on public data to train the student prompt, with private data never being used to train any released prompts, reconstruction attacks are likely not feasible. >**Concerning the DP comparison: The application of DP to both model fine-tuning and prompt tuning is noteworthy. However, even though previous work compare the soft and hard prompt tuning under same DP noise, it is different from model finetuning and prompt tuning. It is hard to ensure a fair comparison between these two approaches, given their inherent differences.** We appreciate the Reviewer's concerns regarding fair comparison and understand the emphasis placed on this issue. However, we want to reassure the Reviewer that using differential privacy (DP) as an indicator of privacy leakage is a standard and widely accepted approach, as demonstrated in numerous studies, including [24] which we referenced earlier. To further support our approach, we direct the Reviewer to Table 2 in [5], where a comparison is made between full fine-tuning, LoRA, prefix-tuning, and other methods, all evaluated at the same privacy budgets of $\varepsilon \in {3,8}$. Additionally, Table 10 in [6] provides a comparison between full fine-tuning and LoRA at a privacy budget of $\varepsilon = 6.7$. These references illustrate that our approach aligns with established practices in the field. >**Overall, your paper demonstrates thorough experimentation and is well-written.** We thank the Reviewer for this positive feedback. >**To further enhance its impact, you might consider addressing the points above, particularly focusing on highlighting the novelty of your approach, expanding the range of privacy attacks examined, and refining the DP comparison methodology.** We hope that the comments above clarify that we have already addressed the novelty of our approach in our submission. However, if the Reviewer has specific suggestions for improving the presentation of our methods and results, we would be more than happy to consider this feedback. We agree that analyzing privacy attacks beyond membership inference attacks (MIAs) is valuable, and we are actively exploring reconstruction attacks. As previously mentioned, using differential privacy (DP) for comparison is a standard practice in the literature, and we have adhered to this protocol to ensure a fair comparison. >**Thank you again for your hard work.** Once again, we thank the reviewer for the constructive feedback on our work. Addressing the comments has significantly improved the quality of our paper. We hope that the answers address the reviewer's concerns and that the scores can be increased.
References
[1] “Bounding Training Data Reconstruction in DP-SGD.” Jamie Hayes, Saeed Mahloujifar, Borja Balle. NeurIPS 2023. [2] “Deep Learning with Differential Privacy.” Martín Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, Li Zhang. CCS 2016. [3] “Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data.” Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian Goodfellow, Kunal Talwar. ICLR 2017. [4] “Scalable Private Learning with PATE.” Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, Úlfar Erlingsson. ICLR 2018. [5] “Large Language Models Can Be Strong Differentially Private Learners.” Xuechen Li, Florian Tramèr, Percy Liang, Tatsunori Hashimoto. ICLR 2022. [6] “Differentially Private Fine-tuning of Language Models.” Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A. Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, Huishuai Zhang. ICLR 2022. (The references below are aligned with the numbers in the submitted paper.) [15] “Flocks of stochastic parrots: Differentially private prompt learning for large language models.” Haonan Duan, Adam Dziedzic, Nicolas Papernot, and Franziska Boenisch. NeurIPS 2023. [24] “DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer.” Junyuan Hong, Jiachen T. Wang, Chenhui Zhang, Zhangheng Li, Bo Li, Zhangyang Wang. ICLR 2024. [27] “Prefix-tuning: Optimizing continuous prompts for generation.” Xiang Lisa Li and Percy Liang. ACL 2021. [31] “Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning”. Haokun Liu, Derek Tam, Muqeeth Mohammed, Jay Mohta, Tenghao Huang, Mohit Bansal, Colin Raffel. NeurIPS 2022. https://openreview.net/pdf?id=rBCvMG-JsPd
Summary
The paper compares private adaptation between closed LLMs and open LLMs, and the authors find that adapted open LLMs always perform better than closed ones at much lower cost.
Strengths
1. This paper presents a comprehensive overview of privacy-preserving adaptation techniques for LLMs. It thoroughly examines various existing methods and compares them across key factors like performance and cost. 2. The results are striking: open-source LLMs consistently outperform their counterparts in virtually every aspect. 3. The paper also discusses the private data flowing and point out that the private data might still be leaked to LLM providers in 3 out of 4 adaptation methods studied.
Weaknesses
My main concern is that the claim that open LLMs are essential for achieving high-quality, privacy-preserving adaptation might be premature. It could be that the field is still new, and we simply haven't yet developed sufficiently powerful algorithms for privacy-preserving adaptation of closed models. I suggest tuning down this claim slightly, acknowledging that the field is relatively new and rapidly evolving, and that the current findings may only be applicable to the privacy algorithms currently available.
Questions
Please tune down the claim by adding a remark that the finding only applies to existing methods.
Rating
5
Confidence
4
Soundness
2
Presentation
3
Contribution
2
Limitations
As above
Summary
This paper compares and contrasts the privacy protections of open vs closed LLMs through conceptual threat models and experimental evaluation. The authors give experimental evidence on local gradient based adaptations performing better than their closed discrete prompt-based counterparts in the areas of privacy, cost-efficiency and performance. The paper concludes that the use of open LLMs not only yields more privacy, but also higher performance at lower costs.
Strengths
The paper is well-written and clearly explains what the contributions are, the background for both DP and LLMs, as well as their experimental evidence. The authors make a compelling case behind the reasons for privacy-preserving adaptations for closed LLMs not being as effective. The problem addressed in the paper is important and the contribution seems novel.
Weaknesses
I think the paper is well-written and don't have any specific feedback.
Questions
N/A
Rating
7
Confidence
2
Soundness
3
Presentation
3
Contribution
3
Limitations
Limitations were addressed
reply to rebuttal
thanks for the rebuttal. please make sure the changes in the rebuttal is integrated into the camera-ready version. I adjusted my score accordingly.
Thank you!
Thank you for your prompt response and for increasing the score, we appreciate it. We had already incorporated the changes into the paper before submitting the rebuttal.
Have concerns been addressed?
We are glad that our updated statements addressed the concerns. Since the rating is still on the borderline, we are more than happy to provide any additional insights if needed.
Thank you and response to rebuttal.
Thank you for the responses, which I have read carefully. I appreciate the effort you put into the general response, and I would welcome it if these new results found their way into the manuscript, especially the results on safety, which I find intriguing and worthy of inclusion in the main body alongside the results on MoE, with the convergence results perhaps being more appropriate for the appendix (although I'm not trying to micromanage you here, this is just an opinion). Some final comments on my side: - On the topic of privacy budgets beyond $\varepsilon=8$: I meant $\varepsilon > 8$. In particular, since reconstruction attacks can be defended against with very large privacy budgets ($\mathcal{O}(1000)$ and above), it would have been interesting to see some results in this direction too. However, as I recognise that I should have written "larger than" and better specified my request, I will not hold this against you, as I also recognise that this is a topic which is tangental to your work. - On the topic of novelty: I agree with you. The contributions of this paper are strong, and my suggestion for the benchmark track were not meant as a criticism. You have my support for the main conference, especially after the rebuttal. - On the topic of privacy units: Thank you for clarifying this. Please specify it precisely in the final manuscript. A side note in case this may have slipped your attention: The argument you seem to be making about using add one since you are using RDP for accounting does not have a causal basis: RDP can be used with any adjacency notion (e.g. using the Google DP accounting library). In conclusion, I'd like to congratulate you on this nice work. I have increased my score to indicate my clear support for acceptance as is.
Response to authors' rebuttal
Thank you for your detailed experiments and explanations. While some of my concerns have been addressed, I still have two main points regarding W3 and Q1 that require further clarification: 1. I appreciate the use of Differential Privacy (DP) as a framework for measuring privacy and providing guarantees. However, I'm concerned about the comparison setting in your experiments. Given that the closed LLM uses Prompt Tuning (PT) while the open LLM uses model Fine-Tuning (FT), there's insufficient evidence to suggest that applying the same DP parameter $\epsilon$ to both PT and FT provides equivalent privacy guarantees for the training data. Therefore, using the same ε across different scenarios may not be appropriate for a fair comparison. 2. The use of DP ε as the sole indicator of privacy leakage level is not fully convincing. As mentioned in the first point, PT and FT may have different privacy risks under the same $\epsilon$. To provide a more comprehensive evaluation of privacy leakage for both Private Training Data and Private Query Data, I suggest incorporating additional attack methods, such as membership inference attacks or model inversion attacks.
Decision
Accept (poster)