Improved Generation of Adversarial Examples Against Safety-aligned LLMs

Summary

This paper introduces two modifications to gradient-guided LLM attack algorithms that can improve its effectiveness and efficiency. First, they skip skip connections in the transformer when propagating gradients. Second, they modify the optimization objective to also include a term for making the latent representations align with those found from a preliminary attack.

Strengths

S1: The two insights that the paper is built on (skipping skip connections and TODO) are clever. I wouldn't have thought of them. This is a good example of a good insight coming from a critical lens on past literature. S2: Figures 2, 3, and 4 are compelling. I'd recommend making the zero line dark in figure 3. S3: Overall, I think that table 1 seems to be a really useful result. S4: If the paper is sound, I can imagine it having some very useful impacts on red-teaming.

Weaknesses

W1: My main challlenge with the paper is the writing. There are a lot of vague, strange, and unexplained phrases like lines 13-16 of the abstract. I think that the writing needs major work. This paper was much harder than a typical neurips paper to understand because of the writing style. Changes should start with the abstract. It took me until finishing section 3 of the paper to understand what the paper's contribution (i.e. what I wrote in the summary field above) really was. W2: The paper lacks qualitative analysis. I would like to see side by side the attacks that went into figure 8. W3: Ideally, this paper would not just work within the harmbench box but would also apply the method to do some form of red teaming that couldn't have been done before.

Questions

Q1: Releasing code seems to be important for this project. What is the update on plans for release?

Rating

5

Confidence

4

Soundness

3

Presentation

1

Contribution

4

Limitations

See weaknesses

Summary

This paper studies and improves white-box suffix-based language model jailbreaking. The authors treats the gradient-based discrete optimization problem involved in jailbreak suffix generation as using a continuous surrogate model to attack the discrete real model. By drawing parallels between this observation and transfer-based image adversarial attacks, the authors leverage and adapt tools from transfer attacks, resulting in an improved version that significantly outperforms the baseline GCG method.

Strengths

1. Language model jailbreaking is a crucial aspect of LLM safety, and the transfer-attack perspective on the discrepancy between the gradient of the loss used in GCG and the actual impact of token substitution is novel. 2. Building on this perspective, insights from two image transfer attacks, SGM and ILA, are applied to discrete token optimization. Comprehensive experiments are conducted to validate the effectiveness and improvements of both methods and their combination. 3. The experimental results are strong. With a substantial reduction in time cost, the proposed method achieves significant improvements in string matching rate and attack success rate under both white-box attack and black-box universal suffix transfer attack scenarios.

Weaknesses

I do not spot a major weakness within the paper. Some minor points are as follows. 1. Regarding the Mistral-7B-Instruct model, Table 1 shows that only the combination of LSGM and LILA surpasses the GCG ASR baseline, whereas for the Llama-chat series, a single component is sufficient (for most cases). This raises my question about whether the previous investigations (as shown in Figures 3, 4, and 5) apply to less-aligned LLMs like Mistral or if they only hold true for better-aligned models like the Llama-chat series. 2. While the proposal is demonstrated to be robust to the choice of $\gamma$ and layer selection $r$, there is no discussion about the impact of the $\beta$ hyperparameter for (LSGM-)LILA$^\dagger$. It would be useful to explore how robust the proposal is to this parameter. 3. The paper could benefit from some polishing, such as correcting the reference to Figure 8, which is currently written as Table 8 in line 333.

Questions

1. A recent study [1] has shown that the refusal of language models can be mitigated by a single direction. It would be fascinating to explore any potential (negative) correlation between the vector $v$ discussed in this paper and the refusal direction presented in the aforementioned work. 2. In addition to the effectiveness of the proposal, previous detection methods [2] found that GCG always generates suffixes with high perplexity. It would be interesting to investigate whether the proposed strategies have any impact on this. [1]: Andy Arditi et al., Refusal in Language Models Is Mediated by a Single Direction. [2]: Neel Jain et al., Baseline Defenses for Adversarial Attacks Against Aligned Language Models.

Rating

7

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

Yes

Summary

The paper explores methods to enhance the effectiveness of adversarial prompt generation based on GCG against LLMs. By leveraging previous transfer-based attack techniques, originally used for image classification models, the authors adapt the Skip Gradient Method (SGM) and Intermediate Level Attack (ILA) to improve gradient-based adversarial prompt generation. The experiment results demonstrate a significant improvement over the vanilla GCG.

Strengths

+ The paper carefully adopts and combines previous transfer-based attacks for image classification, specifically the Skip Gradient Method and Intermediate Level Attack, to target LLMs. This approach significantly increases the attack success rates by over 30% compared to the original GCG. + The time required for attacks based on GCG is also significantly reduced, from 85 minutes to just 3 minutes, while still achieving substantial improvements in success rates.

Weaknesses

Though I believe the method shown in the paper can be applied to other models, the authors should explore LLMs with more diverse architectures to demonstrate the generalization of the proposed method. The three LLMs presented in the paper share the same or similar architecture, particularly in the design of the residual part.

Questions

+ From my understanding, the proposed method specifically refines the gradient for GCG. I want to confirm whether, apart from the gradient refinement, the method for optimizing the tokens remains the same as GCG, i.e., still via Top-k token replacement. If so, I am curious why the time cost for the proposed method is significantly smaller than that of GCG. What contributes to this reduction in time cost? + I am quite interested in the attack success rate when, instead of reducing the gradients from residual modules, we reduce the gradients from the skip connections. Will the ASR still improve significantly, or could it potentially harm the ASR? + When using different decay factors $\gamma$ for the gradient, as shown in Figure 9, did the authors normalize the gradient norms during optimization? I doubt the magnitude of the gradient will also lead to some bias. + Can the universal adversarial suffixes generated by your method transfer more effectively to closed models like GPT or Claude?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The proposed method may depend on the specific design of the LLM architecture. When applying this method to different architectures, it may require more manual effort to tune the parameters effectively.

Summary

The paper takes inspiration from the adversarial attack literature in computer vision to improve the common GCG attack algorithm for LLMs. The paper focuses on the ideas from SGM and ILA in particular. The former enables the author to improve the gradients being used in GCG to be more informative. The latter leads to a new loss that helps optimisation (the idea seems closely related to the work on circuit breaking and refusal directions). The authors validate their ideas empirically on the AdvBench dataset.

Strengths

The paper demonstrates two ways to improve automatic prompt optimisation for LLMs, while the authors primarily evaluate their method to generate adversarial suffixes the insights may also help for prompt tuning more generally. The paper contains several experiments to provide additional insights as to why the proposed modifications help. I am willing to raise my score to acceptance if the weaknesses and limitations are addressed (see below).

Weaknesses

- The empirical evaluation is lacking in parts - Figure 2 & 7 lack error bars making it difficult to assess whether the improvement is statistically significant - The authors are clearly aware of the Harmbench paper as they use the judge provided by this paper, yet the paper uses AdvBench as the source of queries. This is problematic due to the number of semantically similar queries making it difficult to tell if the universal queries are indeed universal. Table 2 should not use AdvBench! - Table 1 and 2 need at least 3 model families, so I urge the authors to also run experiments for those Tables on either Phi or Gemma or Llama3 - Also Table 1 should have at least one jailbreaking method that does not simply generate a suffix, e.g. PAIR. - Figure 4 is missing error bars - Figure 5 is missing error bars - I think it would have been nice to validate Figure 3 & 4 across more models but I understand this is computationally expensive (and the above suggestions are far more important). - The writing could be at times improved significantly. A lot of things are described in words that could be more succinctly and more clearly be explained in math or pseudocode: - the algorithm described in lines 294-307 should have pseudocode. - L(x) should be defined in an equation not text - The abstract is very long and should be shortened. - The paper should discuss continuous attacks [3] in the related work as well as recent work on circuit breaking and refusal directions [1,2] (see also my question on this). I am aware many of these works are too recent to have been included in the initial submission, but I think a added discussion would be valuable to the paper if accepted. - I would have liked to see an experiment attenuating the residual connection gradient instead, as this would strengthen the results from the causal tracing experiment. [1] https://arxiv.org/abs/2406.04313 [2] https://arxiv.org/abs/2406.11717 [3] https://arxiv.org/abs/2402.09063

Questions

The projection loss onto a directional guide sounds closely related to recent work on refusal directions and circuit breaking [1,2]. Could the authors please explain the differences? (And add this discussion to the related work)

Rating

4

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

- No analysis of the perplexity of generated suffixes, which is highly relevant given the ease of implementing a perplexity filter on inputs to the LLM. - The empirical evaluation (see weaknesses). - The paper provides no hypothesis why attenuating the loss from the residual module helps, more precisely why would that gradient have negative cossine similarity with the residual connection.

Response to rebuttal

I would like to express my gratitude for the thorough responses provided by the authors. Regarding the first concern, I acknowledge the improvement compared to GCG under the same hyperparameter setup. I specifically referred to the comparison with GCG*. The methodology presented for Llama2-chat evidently shows improved ASR using just one component from the two algorithmic strategies proposed. This makes me wonder to what extent the observations illustrated in Figures 3, 4, and 5 still apply to other potential LLMs. In the general response, there appears to be an inadvertent replication in Figures V/VII and VI/VIII. I would appreciate it if the authors could confirm whether this phenomenon consistently occurs in both the Mistral and Phi3 models. As for the second concern, the ablation study on $\beta$ convincingly supports the decision to adopt $+\infty$ throughout the paper. Furthermore, it would be beneficial if a quantitative comparison to the refusal direction paper (also mentioned by reviewer heVA) could be provided in future versions. Given the demonstrations and ablations on the effectiveness of the proposed method in improving white-box and transfer ASR, I will keep my current score.

Apologize for the mistakes in Figures VII and VIII

Dear reviewers,   We sincerely apologize for incorrectly presenting the experimental results on Phi-3-Mini-4K-Instruct instead of Mistral-7B-Instruct in Figures VII and VIII. We have conducted experiments on Mistral-7B-Instruct and the results show similar observations to the results of Llama-2-7B-Chat (Figures 3 and 4 in the paper) and Phi-3-Mini-4K-Instruct (Figures V and VI in the PDF).   Best regards, Authors

Correctness about the first "answer" in the rebuttal

Dear reviewer heVA,   We would like to correct some mistakes in the first "answer" of the rebuttal. Specifically: > The results are presented in the attached PDF file as Figures V and VI for Mistral-7B-Instruct, and Figures VII and VIII for Phi-3-Mini-4K-Instruct. In fact, the results of Phi3-Mini-4K-Instruct are shown in Figures V and VI. In addition, in Figures VII and VIII, we incorrectly show the results of Phi3-Mini-4K-Instruct instead of the results of Mistral-7B-Instruct. We have conducted the experiments on Mistral-7B-Instruct and the results exhibit similar observations to those of Phi3-Mini-4K-Instruct and Llama-2-7B-Chat. We would like to express our sincere apologies for the mistakes.   Best regards, Authors

reply

- I would still like to see attacks side by side that went into figure 8. I was referring to looking at examples of prompts. - Overall, thank you for the responses. I think that it would be best for my to keep my score of a 5. Neurips is not allowing authors to submit new versions of the paper (which for the record I think is bad for both authors and reviewers) which means I can't assess the extent to which my main concerns have been addressed.

Answer to Author Response

Thank you for the detailed update. The new figures greatly improve readability of the uncertainty about the proposed modifications. I also second the move toward HarmBench results. As an added bonus, standardization to HarmBench also allows for an easy comparison with the entire set of attacks evaluated in the benchmark. As a smaller remark (a bit too late to test at this point), the hypothesis provided in the response could be tested by porting the gradient modification to other gradient-based optimizers. It might generalize beyond GCG to, e.g. PEZ and autoDan(Zhu) which do not work well on stronger LLMs, especially from the llama famility, due to problems with updating in the computed gradient directions.

Thanks for the rebuttal

Thank you for the detailed responses from the authors. Overall, I am satisfied with the rebuttal but still tend to main my score as 5, and I still have several suggestions: 1. It would be beneficial to include the results for more diverse architectures as suggested by Reviewer heVA in the revision to verify if the observation on skip connections is universal. Regarding the results on Phi3-Mini-4K-Instruct, are these still using the evaluation metric from AdvBench? I would recommend using the evaluation metric from HarmBench instead, since the evaluation in AdvBench is actually quite biased and not accurate. With the current close ASR (~9%) by using the biased metric in AdvBench, I am not sure if the method or the observation on skip connections is indeed universal across different model architectures, or if it is just a specific phenomenon that exists only for one kind of model with a specific training way. Therefore, I would still tend to maintain my score at this moment. If I have overlooked something, please feel free to correct me. 2. I would recommend that the authors also include the results for the GCG with the token replacement settings of GCG-LSGM-LILA, i.e., k=4 and 20, for a complete comparison, which will help readers better understand the effectiveness of the proposed method. 3. Adding the corresponding results above on closed models like GPT-3.5/GPT-4o in the revision would significantly enhance the credibility of the work.