Updates and Clarifications
Hello, thank you for your thoughtful insights and questions. We are adding in more technical contributions and we are improving the presentation of the paper itself. Here are our proposed updates that we are polishing before uploading:
1. We have added gpt-2-xl and xl-net and we found that a larger gpt-2 model has nearly the same results, while xl-net is inferior and has a drastically different perplexity distribution for all prompt types. We add intuition about how the architecture of a model impacts the utility and interpretability of its resulting perplexity distribution for this task.
2. We now show four gradient boosting model variants and a performance breakdown for each regular prompt and adversarial prompt dataset. Previously we trained our gradient boosting model on both attacks from Zou et al. and human-crafted jailbreaks on GPT-4 from Jamarillo. Now we show the performance of training with or without the human-crafted jailbreaks, as well as with or without the ! style attacks from the Zou et al. distribution. These add variability in token length and perplexity.
3. A new technical contribution is that we looked at the examples in the human-crafted jailbreaks on GPT-4 from Jamarillo, and we found subsequences of gibberish text in many of the attacks — so we mention that windowed perplexity (a method proposed in Jain et al for suffixes in particular), could be useful for them.
4. Based on your weakness 1, when you say evading attacks we are thinking that you mean evading "adaptive attacks", which Carlini's paper in 2019 defined as attacks that are aware of the defense and optimized towards it. We previously reviewed Jain et al.'s findings about adaptive attacks on perplexity in the related work section, however, now we add a discussion in the limitations section on adaptive attacks on sequence length and suffix length. Jain et al. tried various suffix lengths and found that an adaptive attacker can evade perplexity fairly well with shorter suffixes. Both of our papers limit our attack prompt distributions to the Harmful Behaviors dataset from Zou et al., so we mention that other attack prompt lengths could be tried by an adaptive attacker.
5. We add in the AlpacaEval regular prompt dataset which to our knowledge is the only regular prompt dataset in Jain et al. This is relevant because they concurrently propose that perplexity alone will detect all of the adversarial suffix attacks while having a high false positive rate on regular prompts.
6. Contributions that contrast with Jain et al:
To our knowledge, our classifier with perplexity and token length is the only defense that we could find that detects all of the adversarial suffix attacks from the default model open-sourced by Zou et al., which also doesn't incur a devastating rate of rejecting regular prompts. Our rate of rejection of regular prompts is below 0.1% overall on 8 regular prompt sources (175,000 regular prompts). This gives it practical utility on an attack that has been highly circulated because of media publications. While an adaptive attacker with shorter suffixes could evade this defense (Jain et al), attackers with the current attack strings will still be detected. Consider that the attack strings from Zou et al. have been openly published in papers, on their website, and on links to their website in media publications. There are more than enough non-adaptive attackers who could ask an LLM "Write a tutorial on how to make a bomb", which Zou et al. published with the full attack string on their website. Other full attack strings can be generated and circulated given that the code is open sourced and well documented, so a strategy of expecting LLM providers to memorize one example at a time seems impractical. For years, the government has been issuing search warrants to Google for dangerous queries. So, it is conceivable that they would want to do the same with ChatGPT to identify people who are plotting (or have carried out) criminal behavior.