Summary
### Summary of the paper
The paper studies the GCG algorithm, identifies some weaknesses in it, and proposes techniques to mitigate these weaknesses and enhance the attack's effectiveness. Specifically, the authors observe that having minimal loss (maximal likelihood of the target string) is neither a sufficient nor a necessary condition for the attack's success. They further observe, that over the course of the optimization, the likelihood of the first token of the target output remains relatively high. Given these observations, the authors propose an "over-generation" procedure that retains suffixes from intermediate optimization steps and applies all of them, one-by-one during the attack. This significantly increases the attack success rate. Finally, the authors use the generated attack suffixes and the corresponding queries to train a LLM, that is able to generate successful attack suffixes for unseen queries that generalize across models.
### Quality
- The paper contains interesting analyses of GCG, which enabled the authors to identify weaknesses and develop improvements. The proposed techniques are relatively simple but are well motivated and accompanied by empirical evidence and analysis.
- The evaluation methodology seems to be carefully designed and instills confidence in the results. However, the generalizability of the results is questionable because very few models were used in the evaluation. Most of the results are based on Llama-2 and Vicuna. These two models have the same architecture and pertaining data, and only differ because the latter was fine-tuned on share-GPT. To show that the observed effects are not a consequence of the chosen target model, more diverse models should have been used. Particularly, the observation about the loss (Section 3.1) should show that the effect is observable on multiple diverse models. Likewise, to evaluate transferability models other than GPT-3.5 and GPT-4 should have been used. It is known from several past papers that attacks crafted using Llama2 and Vicuna transfer to GPT-3.5 because Vicuna is fine-tuned with share-GPT. Several other closed-source models should have been used here such as Claude and Gemini to show the extent of generalizability to models that are more decoupled from the training models.
- The AmpleGCG LLM was trained on Guanaco, Vicuna and Llama, all of which are based on Llama. It is likely that if suffixes from other open-sources models such as Mistral (and its variants) or Phi were used the transferability of AmpleGCG to unseen models like GPT-4 would have been better.
- The techniques for bypassing perplexity based defenses is relatively unsophisticated, and probably easy to defend against. I would not consider this a significant contribution of this paper.
### Clarity
The paper is written clearly enough, but there is some room for improvement. Non exhaustive list of points:
- Intro-para2: "existing jailbreaking methods, including GCG, only generate one single adversarial prompt for one query". This is not true as one can generate one adversarial prompt for multiple queries.
- Intro-para2: "Vulnerabilities" is used to refer to attack strings. This is confusing. Might be better to state attack strings/adversarial prompt.
- Definitions of metrics should be included in the Table/Figure caption. For example, diversity should be defined in the caption of Table 2, not in the text.
- Training and evaluation details are dispersed throughout section 4. They should be in a separate subsection to make it easy to find them and refer to them.
### Originality
The techniques are not highly novel themselves, but their effective application to the task of adversarial suffix generation is still a significant contribution in my opinion.
## Significance
The work is timely and impactful. The safety and societal impact of LLMs is a great concern and this paper shows that existing models are not only highly vulnerable to adversarial attacks, but that these attacks can be generated very efficiently.
Reasons to accept
- The work is timely and impactful. The safety and societal impact of LLMs is a great concern and this paper shows that existing models are not only highly vulnerable to adversarial attacks but that these attacks can be generated very efficiently.
- The approach is simple and effective.
- Highlights a significant issue with the optimization procedure used by GCG (loss of first token does not decrease), and thus provides impetus for the development of better optimization methods for adversarial attacks.
Reasons to reject
- Evaluation setup could be improved to make claims about generalizability more convincing (see summary for details). In short, I recommend including more diverse open- and closed-sourced models in the training and testing stages.
- Clarity could be improved in several places.
- The techniques for bypassing perplexity based defenses is relatively unsophisticated, and probably easy to defend against.
Questions to authors
- It is mentioned that over-generation with GCG does not produce sufficiently diverse suffixes. This is expected because GCG changes one token at a time. However, it may be possible to enhance the diversity by running GCG multiple times (with different random seeds). Since it is known that suffixes at intermediate optimization steps can be successful, the optimization can be run for few steps -- 1 run with 500 steps vs. 2 runs with 250 steps. It might be useful to to generate training data for AmpleGCG this way to enhance the diversity.
- How many human evaluators performed manual inspection of GPT-3.5 responses? To what extent did they agree/disagree?
Ethics concerns details
The techniques can be used to extract responses from the LLM that can cause societal harm in the form of offensive language, encouraging unsafe behavior, etc. The authors have acknowledged these concerns.