ProPILE: Probing Privacy Leakage in Large Language Models

Summary

This paper addresses the pressing issue of potential privacy breaches resulting from the use of large language models (LLMs). The authors propose a novel tool named ProPILE that empowers data subjects, or the owners of personally identifiable information (PII), to evaluate the risk of their personal data being inadvertently disclosed by LLMs. ProPILE allows users to generate prompts based on their own PII to assess potential privacy intrusion.

Strengths

The work is timely and relevant given the current widespread use of LLMs. The idea to empower users to assess potential privacy intrusion using their own data is innovative. This approach could foster a better understanding of privacy concerns among users. The authors showcase ProPILE's utility using the OPT-1.3B model, trained on a publicly available dataset called Pile. They demonstrate how users can assess the likelihood of their personal data being revealed through these LLMs. Importantly, ProPILE can also be used by service providers of LLMs to evaluate their own levels of PII leakage, thus enhancing the robustness of their models against potential privacy violations.

Weaknesses

The application and evaluation of ProPILE are solely based on the OPT-1.3B model. While this provides insights into its performance, it does not necessarily guarantee its effectiveness when used with other Large Language Models (LLMs).

Questions

Whether the authors plan to release any specific datasets alongside their tool, ProPILE. The reference should have been revised. Abbreviations on a title should be capitalized using {}, e.g., PII. Some preprints have been published, e.g., 'Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity' is published on Proceedings of the 60th Annual Meeting of the Association for Computational Linguistic, 2022.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have partially addressed the limitations of their work.

Summary

The paper presents ProPILE, which is a probing tool designed to help identify if the PII of the user can be revealed by probing the LLM-based services. The authors consider two attributes of PII: linkability and structurality, and two types of access: black box and white box. The authors also provide two quantifications: string match and likelihood to quantify the risk of PII leakage. Experiments are done based on these attributes, access, and quantifications to quantify the risk of PII leakage from LLM-based services.

Strengths

• The paper is well-motivated, and studies an important and timely problem. • The authors provide a principled solution to quantify the risk of PII leakage from LLM-based services. • With the given formulation of PII, threat model, probing method and PII leakage quantification, the authors provide extensive study on the PII leakage.

Weaknesses

• The black-box probing is one way to generate a prompt. In many cases, users don't know what PII of theirs are used in LLM and pose privacy threat. o suggestion: establish a prompt database with different kinds of prompt generating rules. These rules are designed for revealing PII. • Exact match may be a strict but narrow privacy definition. The authors should extend the research to a broader concept of privacy. For many entities, there may not be M PII items available. • Also, partial leakage is still leakage, such as one can reconstruct the city but not the street. This broader concept of privacy is missing in this paper, making the scope of the paper limited. • In the meantime, this type of approach, if falls on the wrong hand, can be used to discover PII in LLM-based models. • Density in Fig 3 is undefined. • In the experiment, the exact match percentage and likelihood seems both not high. This gives rise of the question that if the PII leakage problem of LLM-based service severe enough. The authors may want to provide some discussion and insights here.

Questions

See weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See weakness bullet 2, 3 and 4.

Summary

This paper introduces ProPILE, a tool to assess the potential leakage of Personally Identifiable Information (PII) in Large Language Models (LLMs). The work is significant given the recent widespread use and growing concerns around privacy issues of LLMs. The authors demonstrated its use with the OPT-1.3B model and discussed potential societal impacts and ethical considerations.

Strengths

The paper presents a novel tool to tackle an important issue in the current AI landscape: privacy intrusion and PII leakage in LLMs. The development of a probing tool such as ProPILE holds considerable practical significance. The paper provides concrete examples of how the ProPILE tool can be used to detect potential PII leakage in LLMs. The results seem to suggest a reasonable level of effectiveness of the proposed tool.

Weaknesses

The paper acknowledges that the construction of the evaluation dataset involved private information from open-source datasets provided by large corporations. There is an inherent bias in this data collection process, and potential inaccuracies need to be considered when interpreting the results. The paper could have been strengthened by comparing the ProPILE tool's performance with other existing or similar tools/methods for detecting PII leakage in LLMs.

Questions

While the paper introduces the ProPILE tool as an effective strategy, it does not address potential countermeasures that malicious entities might use to circumvent the system. This could be an important aspect to explore in further research.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors acknowledge that the construction of the evaluation dataset exclusively involved the use of private information sourced from open-source datasets provided by large corporations. This approach ensures the ethical acquisition of data. However, they also mention that the data collection process was heuristic in nature, which could introduce a degree of uncertainty or potential inaccuracies. This must be taken into account when interpreting the results.

Summary

The authors propose a privacy attack against large, pre-trained language models to evaluate PII leakage. The idea is that a data subject queries the model using some of their PII to discover whether the model can reconstruct the remaining PII not contained in the query. This attack measures the ability of a model to link PIIs, which is a clear privacy violation. The authors propose a black-box and white-box probing method and demonstrate that an attacker can link PII with a high success rate.

Strengths

* Important problem. Finding tools to empirically measure leakage in LLMs is an interesting and open problem. * Interesting findings. The study finds that pre-trained LLMs can leak PII information meaningfully and that an attacker can be able to link PII with each other even if they only have black-box access to the model. * Clean methodology. I think the paper's method of distinguishing between structured and unstructured PII is meaningful and a good idea. Also, I liked the way that the paper collected PII from the Pile dataset. * Clear figures and well written. The paper is easy to follow.

Weaknesses

* I would have liked to see more ablation studies on the leakage in relation to factors known to increase leakage, such as the duplication rate. A qualitative analysis of the leaked PII could be helpful to understand whether these e-mail addresses are somewhat unique or not. * No promise to release source code. * Limited novelty in the methodology on the privacy attack. The attacks are relatively straightforward. * No discussion on potential countermeasures to prevent leakage. Nits * l. 284: I assume it should say 'small portion of the training _data_'.

Questions

* If I understand correctly, you use regular expressions and NER to extract PII. How did you assess that pieces of PII were linked when extracting from the Pile dataset? * Section 3.3.2 "White-box Probing" is a bit unclear to me: Is the optimization of the prompt conditioned on the PII that you want to reconstruct or is it conditioned on a set of unrelated PII from the same class?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

-

Summary

This paper proposes a novel probing tool to probe potential privacy leakage in large language models. The proposed method includes both black-box and white-box probing. The authors demonstrate its effectiveness on the OPT-1.3B model with the publicly available Pile dataset. The experimental results show that the proposed method can probe the leakage of personally identifiable information (PII) in the Pile dataset, while the white-box probing generally achieves better performance. The results prove that PII leakage could be a serous threat to LLM’s security, calling for more research attention in this area.

Strengths

1. Timely and important topic. With the more and often usage of LLMs, LLMs’ security and potential PII leakage should be taken into account. This paper proposes an early attempt in this area regarding probing PII from LLMs, showing the importance of the research problem. 2. Novel method. The authors propose a novel probing tool including both black-box and white-box probing. 3. Well-written paper. The paper is generally well-written with good presentation quality. I enjoy reading the paper.

Weaknesses

I generally do not find significant weaknesses in this paper. I do have a few small concerns, that, I wish the authors could address in the revision. 1. The definition of the soft prompt \theta_s is not clear (Line 172). 2. Evaluation. In the current form, only one model was included for evaluation. IMO this is fine for demonstration. But if the authors would like to improve it, more models could be considered. A follow-up comment regarding the candidate LLMs: The selection criteria (Line 205-211) should be more clear, and the selection should have its rationale, e.g., is OPT-1.3B the only model that is available for evaluation? If there are more candidates, why OPT-1.3B is the only one chosen? All these details can help improve the paper’s soundness and significance. 3. Lack of examples. It would be nice to add a few examples in the paper to better show the probing results (both black-box and white-box ones).

Questions

1. Is it possible to extend the proposed method to more LLMs or even closed-source LLMs such as ChatGPT or GPT4? 2. How do you justify the choice of OPT-1.3B as the subject LLM for evaluation?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

There are discussions about the limitations and potential social impacts of this paper. I have two more comments. 1. Regarding the limitation, it would be nice to mention the potential generalization ability of the proposed method to other LLMs (or even closed-source LLMs, such as GPT4) 2. Regarding the potential social impact, the authors should directly point out that the research outcome (i.e., the probing tool) might potentially be used for undesired purposes and potentially lead to negative social impact.