Summary
This work proposes an attention-based attack and defense by observing that the attention weights can be correlated to the jailbreak ASR.
Specifically, it first defines three metrics to measure the attention weights: 1. AttnSenwords, which is the average attention weight given to sensitive words across all prompts, timesteps, layers, and attention heads, 2. AttnDepScore, which measures the average ratio of attention paid to input tokens versus all tokens (input + generated) during the text generation process, indicating how much the model relies on the input context when generating outputs, and 3. AttnEntropy, which calculates the average entropy of attention weight distributions across all timesteps, layers, and heads to measure how focused or dispersed the model's attention is across input tokens.
Then, the experiment of four jailbreak methods on two datasets, AdvBench and Dolly, shows ADS and AE are positively correlated to ASR, while ASW is negatively correlated to ASR. Thus, for Attention-Based Attack (ABA), the paper proposes to iteratively refines malicious prompts by minimizing attention weights on sensitive words and maximizing attention entropy through nested task generation, using beam search to select candidates with lowest attention scores until a successful jailbreak is achieved.
ABD defends against jailbreak attacks by detecting suspicious attention patterns through a weighted combination of dependency and entropy scores, then prepends safety assessment instructions to potentially harmful prompts.
Experiments on five models comparing to six baselines demonstrates the effectiveness of ABA. ABD is also illustrated to be effective in defending the baseline attacks as well as ABA.
Strengths
1. This is the first work in analyzing jailbreak attacks through attention mechanisms, providing a fresh approach to understanding how these attacks work at a model behavior level.
2. Comprehensive attack success rates across multiple models, showing strong empirical results on both open and closed source models including the latest LLMs like GPT-4 and Claude-3.
Weaknesses
1. The paper's claimed correlation between attention metrics and Attack Success Rate (ASR) is not well supported by the empirical results. For instance, in Llama2-7B, TAP and DeepInception have nearly identical Attn_SensWords values (0.0089 vs 0.0087) yet their ASR differs dramatically (0.30 vs 0.69), demonstrating that these metrics may not be reliable indicators of attack effectiveness. The authors should address such contradictory examples and provide more rigorous statistical analysis to support their claims about the relationship between attention patterns and attack success.
2. A key concern is that the metrics may be sensitive to differences in prompt structure introduced by different attack strategies (TAP, PAIR, etc.) rather than capturing meaningful patterns specific to jailbreak behavior. To address this, I suggest normalizing each metric by comparing values for sensitive words against non-sensitive words within the same prompt. Specifically:
Normalized ASW = (Attention on sensitive words) / (Attention on non-sensitive words)
Normalized ADS = (Dependency score for sensitive words) / (Dependency score for non-sensitive words)
Normalized AE = (Attention entropy over sensitive words) / (Attention entropy over non-sensitive words)
These normalized metrics would control for prompt-level variations introduced by different attack strategies and show whether sensitive words receive truly distinctive attention patterns.
3. A major limitation of the paper is the lack of concrete implementation details for the core prompt generation and refinement mechanisms. While the authors describe high-level concepts like "semantic-guided scenarios" and "nesting of multiple tasks," they fail to specify how these scenarios are actually generated, how tasks are nested, or what templates and patterns are used. The Refine() function in Algorithm 1 is particularly problematic as it provides no information about how the refinement process works or how it utilizes attention weights. Most critically, there are no examples of how an original malicious query is transformed through their refinement process, making it impossible for other researchers to reproduce or build upon this work.
To improve the paper's contribution, I strongly recommend adding: (1) A detailed description of the prompt generation and refinement mechanisms, including specific rules or templates used; (2) Several concrete examples showing how an original query is transformed through multiple refinement iterations, with attention weights and entropy values at each step; (3) A clear specification of how new scenarios are generated while maintaining semantic relevance to the original query.
4. The adaptability of the parameter alpha to unseen jailbreak strategies is not addressed. The paper should discuss how this critical parameter would perform when encountering previously unseen attack patterns and whether it requires retraining.
5. The authors claim ABA's transferability to closed-source models through "effective scenario nesting templates" learned from open-source models. However, this significant claim lacks empirical validation. The paper should provide experimental results demonstrating this transfer learning capability, or remove the unsubstantiated claim.
6. The definition of the "Queries" metric lacks crucial clarity. While described as "the average number of successful jailbreak attacks between the attack model and the target model," this definition is ambiguous and seems inconsistent with the results. The data shows methods with vastly different Attack Success Rates (ASR) having similar Query counts (e.g., ReNeLLM with 71.8% ASR and ABA with 98.4% ASR have similar Queries of 3.9 and 3.6), which raises questions about what exactly is being averaged. The authors should explicitly specify: (1) whether Queries counts attempts until first success or all attempts including failures, (2) if there is a maximum attempt limit, and (3) how failed attempts are factored into the metric. Without this clarification, the Queries metric cannot be meaningfully used to compare the efficiency of different attack methods.
7. Lack of robustness test. How does ABA perform against jailbreak defense techniques, such as paraphrasing[1], SmoothLLM [2], Backtranslation[3].
[1] Jain, Neel, et al. "Baseline defenses for adversarial attacks against aligned language models." arXiv preprint arXiv:2309.00614 (2023).
[2] Robey, Alexander, et al. "Smoothllm: Defending large language models against jailbreaking attacks." arXiv preprint arXiv:2310.03684 (2023).
[3] Wang, Yihan, et al. "Defending llms against jailbreaking attacks via backtranslation." arXiv preprint arXiv:2402.16459 (2024).
And how does ABD perform comparing with them?
8. The paper could include sections discussing the societal impact and limitation.
Minor:
1. The presentation of tables would benefit from clearer formatting conventions. Column headers should include "(%)" where values are percentages, and table captions would be more informative if they specified the metrics being presented (e.g., "ASR results for different defense methods" rather than "The results of different defense methods").
2. The Related Work section would be more appropriately placed in the main text rather than the appendix, as it provides crucial context for understanding the paper's contributions and positioning within the field.
3. The overall presentation could be improved. For example, the description sometimes is confusing and vague. See questions below.
Questions
1. Regarding attention weight calculation: for a sensitive word at position i, are you measuring:
a) The attention it receives from other tokens (when i is a key position)?
b) The attention it pays to other tokens (when i is a query position)?
c) Or some combination of both?
2. Regarding the timestep dimension T: What is the scope of timesteps in your calculation? Does T represent:
a) Only the positions in the input prompt?
b) Both input prompt and generated output positions?
c) Different ite rations/forward passes of the model?
What is N in eq 2 and eq3? Is it intended to be M because M is defined as the length of input before but never used.
Is padding or truncation applied when calculating all three metrics?
2. In Fig.2, by “the x-axis represents the index of the sentences from the input of the model”, do you mean: 1. each point on the x-axis represents a sentence position within a single multi-sentence input (e.g., sentence 1, 2, 3 from the same prompt)
OR 2. each point represents different input samples/prompts from the dataset.
This distinction is important for interpreting the visualization and should be clarified in the figure description. The current description "the index of the sentences from the input of the model" could be interpreted either way.