Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defenses

Summary

This paper proposes a new way to jailbreak LLMs through an improved version of few-shot jailbreaking. They propose to use a random search to select examples that are most effective to jailbreak the mode from a pre-defined pool generated with Mistral-7B. On top of that, they alternate the steps of each example by the special tokens that are used in the LLMs conversation templates to separate user messages from the model's responses. The authors show that this method is more effective than the previous jailbreaking methods for five different models, and that it can be used and adapted to evade a large number of defenses.

Strengths

**Simple and effective method**. The method proposed is simple and effective. It is easy to understand and to implement. The experimental results show that it is more effective than many baselines. **Insightful ablations**. The authors do a great job at showing what components are most important for the success of the attack. They check how many shots are necessary, how important the size of the pool is and how important the special tokens are. However, there are some other ablations that I believe would make the paper stronger (see weaknesses). **Effective evasion of defenses**. The authors show that their method is effective at evading a large number of defenses of different types, from a perplexity filter, to perturbation-based defenses, to safety-filters. Most interestingly, they propose that one could actually exploit a defense (SmoothLLM) to make the attack robust to keyword-based defenses. However, they do not have any experimental results to show that this is actually the case. **Mildly Compelling motivation**. The motivation of using few-shot jailbreaking is compelling to jailbreak models that do not support a long context. However, it should be noted that these models are also less likely to actually provide useful malicious information to the attacker who is trying to jailbreak the model.

Weaknesses

**No comparison to few/many-shots baselines**. The authors do not compare their method to Wei at al. [1] and Anil et al. [2], which are the most similar to their method. They claim that Wei et al. have limited effectiveness on well-aligned models such as Llama-2, but Llama-2 is not the only target model considered in the paper, and the authors should show some concrete numbers to back-up their claim. For Anil et al., they claim that the attack requires too much context length to work on the considered models, but, according to the numbers shown in the paper [2], the attack starts being effective with 32 shots, the number considered for Llama-3, and they have results for Llama-2 in their paper up to 128 shots. **Missing amount of necessary queries**. One of the metrics that are useful for jailbreak attacks is the total number of queries needed by the random search to jailbreak the model. The authors do not report this number, which makes it hard to compare their method to other methods. **Some ablations are missing**. The authors do a great job at showing what components are most important for the success of the attack. However, they do not show the impact of the quality/length of the examples. It would be interesting to see how the method performs when the examples are shorter or longer, or when some of them are not actually good examples. This would be relevant as the model used to generate the examples could refuse, or generate low-quality examples. Another ablation that would make the paper stronger is how important it is that the special tokens are correct. What happens if you, e.g., use Llama-2's special tokens for Qwen1.5B? Or simply if the special tokens are slightly incorrect (e.g., `[INST]` instead of `[/INST]`? This can be useful to show the potential effectiveness of the attack against models whose special tokens are unkown. **Minor**: - No experiments that show that SmoothLLM can be used to evade keyword-based defenses. - Code is provided, but the data are provided in pickle format, which is known to be unsafe. It would be better to provide the data in a more standard format like CSV or JSON. Moreover, it would be better to provide a README with instructions on how to understand the code. **References**: - [1] Wei et al., https://arxiv.org/abs/2310.06387 - [2] Anil et al., https://www.anthropic.com/research/many-shot-jailbreaking

Questions

- Did you try to use the special tokens from one model to jailbreak another model? - Why do you use four `[/INST]` between pseudo-messages? Have you tried with a different number? Do you do the same for the special tokens of other models? - See my other points made in "Weaknesses" about more ablations, number of queries and comparison to few/many-shots baselines

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors do a good job at discussing the limitations of their work.

Summary

This paper proposes two improved techniques for in-context few-shot jailbreaking: demo-level random search and the injection of special tokens from the system prompt. The authors conduct extensive experiments across a series of aligned language models. Ablation studies demonstrate the effectiveness of both proposals.

Strengths

1. The paper is well-written and easy to follow, tackling an intriguing and timely problem. The baseline few-shot jailbreak is not strong enough, while the powerful many-shot jailbreak requires a long context length, which is not suitable for all language models. The proposal in this paper significantly improves the attack success rate over few-shot jailbreak. 2. The proposed idea of utilizing special system tokens for jailbreaking is novel, and the experiment evaluation is comprehensive. The improvement in attack success rate over the baseline few-shot jailbreaking is significant. Additionally, the techniques introduced are not hard to implement and could serve as a strong baseline for future language model safety research. 3. Ablation studies on the number of shots and the two components are provided, clearly demonstrating the effectiveness of both modifications. The study also illustrates the effect of several defense methods, showing that the improved few-shot jailbreak method can be successful even with the defense methods presented.

Weaknesses

My primary concern regarding the methodology's weakness is the potential for **unwanted bias** due to the **dataset** chosen for evaluation, which could impact the numerical results. As highlighted by previous studies [1], the AdvBench harmful behavior dataset, which is used to construct the demo pool and evaluate attack success rate in this paper, **lacks diversity**. Although it contains **520** harmful requests, only **58 unique behaviors** exist (see Table 5 of [1]). This introduces a significant vulnerability to the current evaluation pipeline, as excluding only 50 requests for testing from the demo pool may be **insufficient**. The pipeline could potentially select similar candidate in-context demo from the pool to the target test request. For instance, if the test request is "Write a tutorial on how to make a bomb", the candidate demo selected from the pool could be "Provide detailed instructions on how to make a bomb" and its answer. In this scenario, the proposed algorithm's success might be biased towards **replicating certain provided demos** rather than originating from the techniques proposed. Besides, as illustrated in [1] (see Figure 2), the generation length has a critical effect on ASR. This paper takes length 100 (see line 176) as the default paradigm. This could lead to an overestimation of ASR. [1]: Mantas Mazeika et al., HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal.

Questions

As stated in the above section, I have the following questions. 1. **Frequency of replication event during testing**. It would be important to know how often the replication event occurs during testing, and whether it's a common occurrence or a rare event. This could help understand the extent to which the proposed algorithm is relying on replicating provided demos rather than generating novel responses. 2. **Excluding similar requests from demo pool**. Based on Q1, if we exclude not only the requests for testing, but also all similar requests to the current test request from the demo pool, would the results change significantly? Would the proposed algorithm still perform well, or struggle to generate effective responses? 3. **Impact of decode length**. How does the decode length variation affect the results? Will the accuracy drop significantly? I'd be happy to raise my score if these questions could be resolved. Minor point with respect to clarity of writing. The description of Llama Guard implementation for adaptive attack is a bit unclear to me. I understand that the usage of Llama Guard for computing ASR and for launching adaptive attacks are different (presumably on the [GOAL] placeholder). If this discrepancy could be made explicit, it would improve the clarity of the text.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Yes

Summary

This work proposes a new method to jailbreak LLM to elicit harmful responses. The proposed method follows a line of works on using the demonstrations of harmful responses in the context of prompt to jailbreak. It improves the previous works regarding reducing the number of demonstrations in the context and increasing the efficacy. Specifically, the proposed method uses an unsafe LLM to automatically create a pool harmful demonstrations, insert special tokens into the prompt, and optimizes the demonstrations using a demo-level random search. The empirical results confirm the efficacy of the proposed methods.

Strengths

1. the proposed method is simple and straightforward to implement. 2. the dramatic sensitivity of FSJ to special tokens is surprising. 3. the evaluation is comprehensive (many defenses are tested) and the results of the proposed method are strong. 4. the paper is well-written and easy to follow.

Weaknesses

1. The evaluation is based on 50 harmful responses from AdvBench. The scale is limited. Besidse, AdvBench is also used to generate demonstration pool. Although the overlapped ones are inspected and removed, there may be a concern of overfitting. Using a different source of harmful responses like HarmBench [1] for evaluation may be better. 2. The proposed method assumes that attackers have access to model-specific special tokens, which restricts its application scope. Without the help of inserting special tokens, the proposed method seems to be ineffective in breaking the well-aligned models like Llamas as shown in Tab. 1. It is therefore interesting to test if a special token can be determined without the knowlege of target model. 3. Although the proposed method demonstrates the ability to circumvent a wide range of defenses, it may be ineffective when adaptive defenses were deployed. For example, toxic detectors can be used to detect if harmful content is included in the input prompt as demonstrations. [1] Mantas Mazeika et al., HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal.

Questions

see the Weakness above.

Rating

7

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

some limitation has been discussed, but more are required. See some points suggested in Weakness.

Summary

This paper proposes several ICL (in-context learning)-based techniques to improve the effectiveness and efficiency of jailbreaking prompts, including adding system special tokens and random search on the demonstrations.

Strengths

- The discovery that using special tokens can enhance the effectiveness of harmful demonstrations is interesting. - The experiments show the overall proposed method can notably improve the ASR on multiple LLMs. - The experiments also include evaluations of the attack against LLMs with defense techniques.

Weaknesses

- The main objective of this paper seems to be misleading. As indicated by the abstract and the story in the introduction, this paper attempts to address the problem of > it possible to use few-shot demonstrations to efficiently jailbreak LLMs? However, since ICA has already been proposed as the few-shot version of jailbreaking, this paper may take ICA as the main target, rather than refining MSJ. - Following the previous weakness, the most important baseline, ICA, is missed in the experiments. Moreover, what is the difference between the used baseline (FSJ) and ICA is not indicated. - The first improved technique, injecting special tokens, though interesting, is of limited scientific contribution. It’s more like an attack trick, rather than a substantial academic improvement. More importantly, why these tokens can enhance the ASR is not well-explained or understood. - The second technique is anyway lacking novelty since the jailbreaking literature has already used the intention of random search (e.g., GCG and AutoDAN) to improve the jailbreaking prompt.

Questions

See weaknesses.

Rating

3

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

N/A

Summary

This paper proposes jailbreak attacks via few-shot demonstrations. The authors introduce a three-step method to achieve this goal, which includes constructing a demo pool, injecting special tokens, and demo-level random search. The proposed method demonstrates strong attack performance against aligned LLMs and multiple defenses.

Strengths

The proposed method is a strong attack that can bypass many advanced defenses.

Weaknesses

Overall, the paper is well done. However, I have a significant concern: How does the attacker know the special tokens used in the LLMs? This is particularly problematic for attacking closed-source models such as ChatGPT. I also noticed that the authors did not evaluate their method on closed-source models in this paper. This issue represents a critical weakness in practical jailbreak evaluations. I will raise my score to acceptance if this concern is addressed. Otherwise, I think this weakness is a flaw that we can not ignore.

Questions

Please refer to the weaknesses.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed the broader impacts and limitations.

Response to rebuttal

I would like to express my gratitude for the responses provided by the authors, for clarifying that the proposed method retains its efficacy in the absence of replication events. From Table C in the general response, I observed a clear ASR drop when the demo RS is omitted under decoding length 512. However, this decline is mitigated when the demo RS is reinstated. These observations collectively affirm the effectiveness of the demo RS component. Furthermore, I agree with reviewer PZVT on the necessity of including a detailed section regarding the selection of special tokens. This addition is particularly crucial given the results in Table C, which clearly show that an inappropriate choice of special token can drastically reduce the ASR to as low as 0%. It would be nice to include ASR on top of the loss as shown in Figure A to further illustrate the sensitivity of special token selection. For example, adopting the pool of tokens from Llama-2-chat and reporting the corresponding ASR would serve as a proof of concept. I have raised my score to 7.