We thank the reviewer for engaging in discussion with us and raising their score!
As requested, we are providing a further breakdown into the results and PII provided by users in Wildchat. We used GPT-4 to annotate 5000 conversations, asking it to annotate the requested task, if there is personal information provided, and to provide the type of the personal information (as also described in the paper). We find that 21% of the queries include what is identified as sensitive information by GPT-4, however, filtering by task to throw out fictional cases (there is instances of role-playing, creative writing, etc.) omits 31.3% of these cases, leaving an overall of 14.5% of queries that include what can be considered as ‘real PII’, but we do acknowledge that these could still include some PII that is not sensitive/of real users. To further analyze these, we sorted the queries including personal information by their tasks, and found that text editing or writing tasks (CV editing, letter/email/statement generation) do overall contain the bulk of PII (34.0% of the non-fictional sensitive queries), which include the authors name as expected, for instance:
```
...my cv:\\n\\nIbrahim A Ibrahim Gaddari\\tFLT 10, Tropea Court, Triq in-nazzarenu \\n\\t(+356) 9974 5663\\n\\t\\n\\n\\nASPIRATIONS\\nSeeking the right corporate environment within which I may gain practical experience...
```
One surprising category with PIIs is the task ‘translation’ (6.6%), as people tend to mindlessly copy and paste their personal information as part of a sequence to be translated, here is an example:
```
….Letter of Guarantee from Daughter This letter is to confirm that I, Zeqi Qian, am the daughter of Qunlei Qian and that I have invited my father to visit the UK as a tourist. I will begin my course in Engineering Science as a first-year student at Oxford University in October. My passport number is EJ6489540, and my student visa number is 022531800….
```
Apart from translation, another surprising common category of queries that involves self-disclosure of PIIs is code editing (20.4%), as people tend to dump their code with their name, email address and API tokens in comments to chatGPT:
```
import Optional from aiogram import types API_TOKEN = '6084658919:AAGcYQUODSWD8g0LJ8Ina6FcRZTLxg92s2w'
```
We believe that such high, inadvertent disclosure rates are of significant importance for researchers and data curators to be aware of, and call for design of appropriate nudging mechanisms in services such as ChatGPT. We will add these case studies and the fine-grained analysis to the paper. Please let us know if there is any further information we can provide!