Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory

Thank you for the positive endorsement on this new direction of investigating contextual privacy implications in LLMs.   ### Scaling of the benchmark Our benchmark could be expanded in size by adding more variables to the factorial vignettes (i.e. the underlying templates) for each sample. For example, the set of information types in Tier 2 can be expanded to include social media handles or zip code. Theory of mind scenarios can also be expanded by populating causal templates (Gandhi et al., 2023), and expanding on the contextual factors. However, we are confident that **the core message of our paper regarding the importance of context when dealing with privacy in language will very likely not change**, no matter how many new scenarios are added to the benchmark. We will add this discussion to our paper.

We appreciate your strong positive feedback that our contribution is significant and original. ### 1. Metric interpretation and definition We thank the reviewer for raising this issue, we agree that including this information will improve the readability of the paper. We will include the following table in the revision of the paper to clarify and categorize all of the metrics and the values they take. Below is a summary of this table: | Tier | Metric Name | Metric Type | Range of Value | Interpretation | Figures and Tables | | --- | --- | --- | --- | --- | --- | | 1, 2.a and 2.b | Sensitivity Score | Score | [-100,100] | Lower means more sensitive | Figure 2, Table 2 | | 3 & 4 | Leakage | Rate | [0,1] | Lower means less leakage (better) | Tables 4&5 and Figures 3&4 | | 3 | Information Accessibility Error | Rate | [0,1] | Lower means more accurate (better) | Table 4 | | 3 | Binary Control Question Error | Rate | [0,1] | Lower means more accurate (better) | Table 4 | | 4 | Omits Public Information (Utility) | Rate | [0,1] | Lower means more accurate (better task utility) | Table 5 and Figure 4 |   ### 2. Manual inspection of LLM responses Following your suggestion, we conducted a thorough manual review of all 200 GPT-4 responses for the action item generation task in Tier 4. We discovered 16 instances where private information was leaked, yet not detected by exact string matching. For instance, the private information was "move to VISTA," but the response mentioned "transition to the VISTA project". This indicates overestimation of the models’ performance in Tier 4. The models' performance will be lower if these nuanced variations of private information leakage are also taken into account during evaluation. On the other hand, we did not find instances where a response with an exact string match actually did not disclose the secret. We will add this discussion in the revised draft.   ### 3. Testing GPT-4 on “ChatGPT-generated data” vs. “GPT-4 generated data” We would like to thank the reviewer for raising this very interesting issue, and want to clarify that the scenarios of Tier 3 and Tier 4 are generated by GPT-4, however, they have received human intervention to make sure the stories are sound and serve the purpose of the tier. To address the valid issue raised by the reviewer, we re-generated the Tier 4 scenarios with ChatGPT, and evaluated GPT-4 and ChatGPT on it and compared the results with those from GPT-4 generated Tier 4. You can see the results below. | ChatGPT results | Action Item | | Meeting Summary | | | --- | --- | --- | --- | --- | | | Mean | Worst | Mean | Worst | | GPT-4-generated | 35.0 | 70.0 | 65.5 | 95.0 | | ChatGPT-generated | 37.5 | 85.0 | 53.5 | 85.0 | | GPT-4 results | Action Item | | Meeting Summary | | | --- | --- | --- | --- | --- | | | Mean | Worst | Mean | Worst | | GPT-4 generated | 32.5 | 75.0 | 38.0 | 90.0 | | ChatGPT-generated | 28.0 | 65.0 | 38.0 | 85.0 |   GPT-4 still outperforms ChatGPT with a significant margin on the ChatGPT-generated Tier 4. Moreover, GPT-4’s scores improved in comparison to its results on the GPT-4-generated Tier 4. Conversely, ChatGPT’s performance change was mixed. It showed improvement in the action item generation task, but showed a decline in its performance on the meeting summary task. To summarize these results, our **findings that GPT-4 outperforms ChatGPT in terms of privacy leakage still holds, even in the case where the meeting script is not generated by GPT-4.** We will also add additional discussion on the limitations of using LLMs for synthesizing evaluation data, and include these results in the appendix of the paper.   ### 4. Background of the human annotators We unfortunately did not collect background information on the human annotators on MTurk, however, the 2016 law study by Martin & Nissenbaum (which we use as backbone for Tiers 1 and 2 and show to have high correlation with our annotations), collected the Turkers’ gender, age and privacy concerned-ness (the ‘westin’ privacy categorization which determines how much a person cares about privacy in general). Their study show that *factors such as gender, age and privacy-categorization (i.e. privacy concernedness) of the annotators has no statistically significant correlation with their privacy preferences*. Additionally, they show that contextual factors in the vignettes (the actor, information type and use-case) can better explain the privacy preferences (pages 211-212 of the study). We will add this to the discussion section and thank the reviewer for raising this important issue.

### 5. Statistics of our benchmark We included the number of examples for each tier in Section 3. However, we will also include the following table in the revision of the paper for better readability. | | Tier 1 | Tier 2 | Tier 3 | Tier 4 | | --- | --- | --- | --- | --- | | Total number of samples in the tier | 10 information types | Tier 2.a: 98 vignettes, Tier 2.b: 98 vignettes | 270 scenarios | 20 transcripts | | Source | Maden (2014) | Tier 2.a: Martin & Nissenbaum (2016), Tier 2.b: GPT-4 generated text | GPT-4 generated texts with expert intervention | GPT-4 generated texts with expert intervention | | Total number of questions | 10 sensitivity questions (multiple choice) | 98 * 2 privacy expectation questions (multiple choice) | 270 Response generation question (free-form), 270 Information accessibility questions (list), 270 Private information sharing understanding question (list), 270 control questions (binary) | 20 Action item generation questions(free-form), 20 Meeting summary generation question (free-form) |   ### 6. Details of the tested LLMs Thank you for the suggestion. We will update them in the revision.

We thank the reviewer for their positive endorsement on our interesting discoveries and our investigation of the fundamental gaps in the privacy analysis of LLMs. ### 1. The example in the introduction for the “Contextual Integrity Theory” The ethos of the theory of contextual integrity is that whether or not an information type is private relies on the context, and not just the information type (Nissenbaum 2004). As for the example, medical history is actively shared in many contexts, such as between insurance companies, between partners and family members and with different physicians and hospitals (HHS 2021), so passing judgment on sharing medical history is more nuanced than having a static rule of never sharing something. Further examples are provided in the wikipedia entry of contextual integrity (https://en.wikipedia.org/wiki/Contextual_integrity): Sharing one's SSN with the IRS for tax purposes is not a privacy violation, but sharing it with the local newspaper is. We kindly ask the reviewer to please let us know if there are further questions on this! **Reference** US department of health and human services (HHS) https://www.hhs.gov/sites/default/files/ocr/privacy/hipaa/understanding/consumers/sharing-family-friends.pdf   ### 2. Reference for “Theory of Mind” in the introduction. We will add the reference to Premack & Woodruff (1978), which we have included in Section 2, to the introduction as well.   ### 3 & 4. LLMs understanding of privacy and their performance on our benchmark Answering the question of how LLMs perceive privacy is the overarching theme of the paper. Privacy is not 0-1 nor clean-cut, as explained in depth by Brown et al. 2022 (mentioned in the paper as well), there are many layers of nuance to what is considered private. That’s why we design a multi-tiered benchmark, to disentangle different levels of ‘understanding privacy’, starting from having the raw knowledge of what sensitivity and privacy are (Tiers 1 and 2), and adding complexity in terms of using and applying the understanding of these concepts (Tiers 3 and 4). As the results show, the performance of the different models varies heavily across different tiers of our benchmark. For the open-source models (Llama2 70B , Llama2 70B chat and Flan-UL2), we see the worst performance on all tiers, starting with low agreement with human annotations in tiers 1 and 2 (Table 1). This shows that this group of models is not aligned with human preferences and lacks the knowledge of human expectations of privacy. Moving to tier 3 (Table 4), we can see that the open-source models have high error on the information accessibility questions, showing that they lack theory of mind and cannot keep track of who has access to what information, so that is another cause of their poor performance. This also explains the failures of such models on Tier 4 (Table 5) The situation is more nuanced for OpenAI models, specially ChatGPT and GPT-4, as the have high agreement (correlation) with the human preferences for the first two tiers, indicating that they have the knowledge of human preferences, however, they cannot reason over it and operationalize it, which is why we see failures in Tiers 3 and 4 for them. We like to emphasize that the failures we observe cannot be boiled down to theory of mind failures, as for GPT-4 we observe in Table 3 that the information accessibility questions and the control questions have very low error rates, demonstrating that the model has kept track of who knows what, but it cannot reason over it and operationalize it at generation time, which is why we see leakage of secrets. If the reviewer’s question alludes to whether or not the models are able to follow our instructions about privacy, then this falls more under the realm of RLHF and instruction tuning, rather than privacy directly. Also, we follow the methodology that is commonly used in literature for benchmarking LLM capabilities (Bulian et al. 2023, Birkun & Gautam 2023), where we ask the models direct questions about situations and collect their response, to empirically study wether they understand different concepts. Finally, regarding your question 3.b, Since only X and Y shared the information and Z did not, the models should not mention Z's name when they are asked to list the characters who shared the private information. We will clarify this in the revision. **References** Bulian, Jannis, et al. "Assessing Large Language Models on Climate Information." arXiv preprint arXiv:2310.02932 (2023). Alexei A Birkun and Adhish Gautam. 2023. Large language model-based chatbot as a source of advice on first aid in heart attack. Current Problems in Cardiology (2023).

### 5. Why contextual privacy is important especially in real-world applications The framework of contextual integrity was first introduced in 2004 and further developed in the 2009 book ‘privacy in context’, and its importance is underlined by its broad application and adoption, as there is thousands of work building on it, and an annual symposium on privacy through contextual integrity (https://privaci.info/). As for the importance of contextual integrity in language, there is several prior work discussing the importance of context in privacy for language (Shvartzshnaider et al. 2019 and shi et al. 2021), nominally the work by Brown et al. (2022) which accentuates the nuances and complexities of privacy in language. The aim of our work is to fill this gap that has been pointed out by prior work multiple times especially as more LLMs are getting deployed in real-world applications. That’s the reason why we designed Tier 4, where we focus on real-world applications of LLMs, such as meeting summary generation and personal action item generation, similar to the services deployed in Microsoft Teams (Ajam 2023).   ### 6. The difference between existing works in machine theory of mind and ours Our work is the first to explore the intersection of privacy and theory of mind in AI. While there are many existing studies on machine theory of mind, as detailed in the most recent survey paper, none of them have studied the relationship between privacy and theory of mind. Also, our Tier 3 and 4 are not subsets of theory of mind experiments for LLMs. Instead, theory of mind is a distinct component of those tasks. In this context, theory of mind refers to the capability to infer and reason about mental states in scenarios where information is asymmetric. The goal of Tier 3 and 4 is to investigate privacy implications (e.g., controlling information flow) of LLMs, which requires this capability to understand and assess who knows what.   ### 7. Summarization of our contribution Our contributions can be summarized as the following: We operationalize the notion of contextual integrity to fit the use-cases of large language models. We introduce a multi-tiered benchmark building on this notion, where we study inference-time privacy in interactive scenarios, which has not been done before. Our benchmark evaluates different nuances of privacy, on a wide range of models, including knowledge of sensitivity in different contexts, reasoning and theory of mind. We will add this to the introduction in the revision of the paper.

Thank you for the positive feedback on the strong foundation of our approach in contextual integrity theory and positive endorsement of our experiments. ### 1. The scope of the study While our benchmark probes models for their privacy reasoning capabilities through the theory of contextual integrity, judgments on what should and should not be revealed can be deeply intertwined with the moral and cultural aspects of an interaction. Social psychologists have studied the moral incentives behind why people might reveal each others’ secrets, as a form of punishment (Salerno & Slepian, 2022), and we encourage future work to further study such incentives in language models more extensively. Moreover, there is another less discussed aspect of human-AI interaction: people may feel more at ease sharing information with AI models --- information they would hesitate to share with other humans --- believing that their disclosures will remain secure. This encompasses different topics, from personal matters to corporate confidential documents (Park 2023). These incentives could also impact secret keeping of LLMs and require further studying.   ### 2. Results in Section “3.5 Human Annotations” The correlation of 0.85 is observed between the original human annotations of Martin & Nissenbaum (2016), which was conducted using Mechanical Turk as well, and the human annotations collected for our paper. We include this metric to show there is consensus between our findings and that of prior work (in technology law), for human preferences. We will clarify this in the updated draft.