Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes

Summary

This paper analyzes the loss landscape of the refusal loss in the jailbreaking problem and proposes a method to detect jailbreaking input. This method has two steps: 1) sample multiple outputs and vote on the results, and 2) take the gradient with respect to the input and measure the gradient norm. Overall, the proposed methods are effective.

Strengths

- The proposed methodology can effectively detect jailbreaking input. - The paper is well-organized and the overall writing is good. - The evaluation covers a number of attacks and adaptive attacks.

Weaknesses

- **The main weakness** is that the proposed method requires $N · (P + 1)$ queries. In the paper, the authors select $N=P=10$ , which means it requires more than 100 queries for one sentence. Even though the authors claim that one can use batch inference, the memory cost makes the proposed method unrealistic in practice. Step 1 of the proposed method is trivial and not sound due to its computing efficiency. The paper does not compare the time and memory efficiency with other baselines. - The presentation of the experimental result is not good. The paper puts Figure 2 and Table 1 instead of the full results (Table A1 and Table A2) in the main body. I would suggest combining Table A1 and Table A2 and putting them in the main body. Additionally, Figure 2 is a bit unclear and insufficient, e..g., what the blue line indicates? - Equations 4 and 5 are not very necessary because the message is just that the authors add the same perturbation for all embeddings. Theorem 1 is somehow redundant. - The considered jailbreak keywords are much less than those used in GCG's paper.

Questions

- In sec 4.5 Utility Analysis, the author compares Vicuna-7B-V1.5 & Vicuna-7B-V1.5 with Gradient Cuff, why the metric is 'Average answer accuracy' instead of 'FDR'? What we actually care about is whether a model will provide an answer to a benign sentence, if both models provide output, then such outputs should be the same. What is the meaning of ``By setting a 5% FPR on the validation dataset''? - Could the author compare or discuss the proposed method against [2,3,4]? - Is it possible to apply the proposed defending method in closed-source GPT? - How to sample in a batch with the same temperature? - Why the considered jailbreak keywords are much less than those used in GCG's paper [1]. ### Ref * [1] Universal and Transferable Adversarial Attacks on Aligned Language Models * [2] Safedecoding: Defending against jailbreak attacks via safety-aware decoding * [3] Single-pass detection of jailbreaking input in large language models * [4] RAIN: Your language models can align themselves without finetuning

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

Currently, I don't see much clear discussion on the limitations even though the authors claim they are in sections 4.4 and 4.5. The discussed limitations hold for all jailbreaking defenses instead of only the proposed methods.

Summary

This paper introduces the concept of refusal loss function for detecting language model jailbreaking attacks. The refusal loss is defined as $1$ minus the refusal rate, and it is observed that on malicious instructions, the refusal loss tends to have a smaller value and a larger gradient norm. Based on this observation, a two-stage defense algorithmic framework called "Gradient Cuff" is proposed to detect jailbreaking attempts by setting thresholds on these two quantities. Experimental results on Vicuna-7b-v1.5 and Llama2-7b-chat demonstrate the effectiveness of the proposed method against baseline defenses, including the perplexity filter, the erase-and-check method, SmoothLLM, and Self Reminder.

Strengths

The paper's contribution lies in the investigation of the refusal loss landscape, which is a novel and intriguing aspect. While the definition of the refusal loss itself is not new, the exploration of its properties and behavior in the context of language model jailbreaking attacks is original and valuable. The proposed Gradient Cuff method has several advantages over existing baseline defense methods. Firstly, it can detect jailbreak attacks in multiple forms, whereas methods like the perplexity-based filter are mostly effective against suffix-based attacks. Secondly, Gradient Cuff allows for direct control over the false positive rate (FPR) versus true positive rate (TPR), which is not easily achievable by most existing defense methods. This feature provides a useful trade-off between detection accuracy and false alarm rates. The ablation experiments conducted in the paper are comprehensive and effectively illustrate the usefulness of the two proposed stages separately. The comparison against baseline defenses also showcases the advantages of Gradient Cuff over existing methods. Overall, the paper presents a promising approach to detecting language model jailbreaking attacks and contributes to the ongoing efforts to improve the safety and security of language models.

Weaknesses

My concern of weaknesses are as follows. I. A discrepancy between the 2-stage introduction of Gradient Cuff in the main text and its description in Appendix A.5, where an additional 3rd step involving sampling and rejection via the JB function is considered. This operation caught my attention because (1) it implies an extra layer to the introduction of Gradient Cuff in the final step, and (2) no ablation on the JB function was provided to study its impact on Gradient Cuff's performance. Prior research (e.g., [1], Figure 2) has demonstrated the instability of string matching, which is the JB function used throughout the paper. Therefore, an ablation on this component (if implemented throughout the paper as the 3rd stage of Gradient Cuff for TPR computation) should be taken into consideration. II. The experimental setup is confusing across different tables and figures. In line 229, it is stated that σ (FPR) is set to 5%. However, the full result of Figure 2 (Table A1) contradicts this claim - one could also tell this from Figure 2 directly. This also makes the comparison in Table 3 invalid, since the numbers appearing without adaptive attack do not have σ (FPR) as 5%. The numerical description in line 259 on improvements is also invalid. III. Llama2-chat system prompt usage. In section 4.1, it is claimed that "As for the system prompt, we use the default setting provided in the fastchat repository." However, fastchat does not provide the default Llama2-chat system prompt by default (see https://github.com/lm-sys/FastChat/pull/2162), which is **not common** for jailbreak evaluations (see e.g., [1] [2] [3]). This is partially supported by the numbers in Figure A1 (a), where the GCG attack success rate on Llama2-chat is 0.694, which is significantly higher than reported in previous works [1][2]. [1] Mazeika et al, HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. [2] Zou et al, Universal and Transferable Adversarial Attacks on Aligned Language Models. [3] Chao et al, JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models.

Questions

I have the corresponding questions following the weaknesses section. I. How much does the 3rd JB indicator layer functions for Gradient Cuff, and what happens if we ablate this layer with other indicator candidates (e.g., with Llama Guard)? Will Gradient Cuff still offer great advantages, or the most contribution will be attributed to the JB filter? II. Should the FPR in Figure 2 and Table A1 be exactly 5%? III. What happens to Table 2 if the correct Llama2-chat system prompt is used? Do we still see significant increments over TPR when we incorporate stage 2 of Gradient Cuff into consideration?

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

Yes

Summary

The authors propose Gradient Cuff that uses the gradient norm for response y being a non-refusal (using their refusal loss) towards the prompt x to detect jailbreak attacks. Since the designed loss is non-differentiable (along with the generation process), the authors propose to use a zeroth order gradient estimate. Here the gradient is estimated via multiple evaluations using slightly perturbed input embeddings. The authors show that Gradient Cuff is increasing the refusal rate significantly for state-of-the-art jailbreak attacks while maintaining reasonable utility.

Strengths

1. The adaptive evaluation makes clear that the increased refusal rate is meaningful 1. The authors highlight that for many prompts resulting from jailbreak attacks, (a) the attacked model sometimes refuses and sometimes answers to the request and (b) the refusal loss gradient towards the adversarially crafted prompt is of large norm if the attack was successful 1. The method Gradient Cuff is simple yet effective 1. The false positive rate of Gradient Cuff is expected to be low in many domains since most LLMs only rarely respond with phrases like "I am not able to" etc.

Weaknesses

1. The defense may be considered costly. In the worst case, the Gradient Cuff requires N (P + 1) model evaluations (10 * 11 = 110 in the experiments). Although this cost is somewhat mitigated by a two-step procedure. The argument of batching is somewhat questionable, considering that the LLM operator could batch multiple user requests otherwise. 1. Only two LLMs are evaluated that are both Llama-based. It would make the paper more convincing if the gradient phenomenon is shown to hold for a large number of LLMs. 1. The utility evaluation is rather shallow, only focusing on multiple languages.

Questions

1. What dataset is used for tuning the false positive rate? How well does this parameter generalize to slightly different settings? Can the authors also report, e.g., AUC scores for independence of a specific threshold? 1. What is the reason for the (costly) zero-th-order gradient estimates? I imagine that one can also use backprop at each generation step that yields certain jailbreak keywords (token), using the log-likelihood of this keyword (token).

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Limitations are sufficiently addressed

Summary

In this paper, the authors proposed Gradient Cuff, a novel jailbreak detection method based on the refusal loss landscape. Specifically, 1) the refusal loss is defined as the probabiliy for the LLM to generate a refusal as response; 2) the authors further made the observation that the refusal loss tend to be more precipitous for malicious queries than benign queries. Accordingly a two-stage detector surfaces: 1) rejecting the queries with the refusal loss less than 0.5; 2) rejecting the queries with the gradient norm of the refusal loss larger than a threshold. In practice, Gradient Cuff perfoms multiple independent probabilistic inferences on a query to estimate refusal loss as the frequency to observe refusal related keywords in the reponses; it uses zero order estimation to approximate the gradient with random purtabations to the input token embeddings; the natural refusal rate is measured with knowingly benign queries to facilitate the decision of the gradient norm threshold. This detection methods was evaluated together with baseline methods like Smooth-LLM and Self-Reminder on a subset of the AdvBench dataset agaisnt various jailbreaking attacks including AutoDAN and PAIR. Results showed that Gradient Cuff yields both a higher refusal rate for malicious queries and lower refusal rate for benign queries, indicating that it provides better defence. Adaptive attack experiments are also done where the attacks are launched when Gradient Cuff is already in place. The results were also in favor of Gradient Cuff.

Strengths

+ The authors did a good job illustrating the refusal loss with mathematicl formulations as well as plots. + The paper comes with a proof of the error in the gradient norm estimation as well as a verification through empirical results.

Weaknesses

Gradient Cuff requires actually analysing the harmfulness of the generated response. Although the authors claim that the overhead due to Gradient Cuff can be even lower than some of the baselines, it is still a significant difference, so the comparison with them are not entirely fair.

Questions

+ As Gradient Cuff directly analyzes the harmfulness of the response to the potentially malicious query, isn't the maliciousness detection methods like LlamaGuard the more suitable baselines? + What is the trade-off with utility for the baseline methods? How to verify that Gradient Cuff reaches a better trade-off?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors discussed the limitations in Section 4.4 and 4.5 regarding the utility degradation and performance degradation when encountered with adaptive attacks. The broader impact is discussed in Section 6 saying that no negative impact of Gradient Cuff has been foreseen.

Response to rebuttal

I thank the authors for the detailed explanations and the effort in running additional experiments! Most concerns are resolved; however, the high computational cost remains. I am not doubting that batch processing is possible or may yield gains. However, whoever runs the model will ~100x their compute cost. Instead of batching the same request for gradient cuff, whoever runs the model, could otherwise batch different prompts/requests instead.

Response to rebuttal

I would like to extend my gratitude to the authors for their clarifications and for conducting additional experiments. I apologize for my previous misunderstanding regarding the FPR. I appreciate the updated results demonstrating that Gradient Cuff continues to hold its advantages over the (newly included) baselines, even as the jailbreak judge varies. At this moment, I have no significant concerns. After reviewing the rest of the rebuttal and the general response, I agree with reviewer GFwB that while batch inference is beneficial, the computational cost of deploying Gradient Cuff could impact its practicality. Given the effectiveness of the proposal across multiple LLMs and the aforementioned limitation, I have raised my score to 5.

Thank the authors for the explanation

Thank the authors for the explanation. I have read the comments and rebuttal, I will maintain the score due to the main weakness of using $N*(P+1)$ queries.