Tree of Attacks: Jailbreaking Black-Box LLMs Automatically

Summary

Note: I have seen this paper before, and I have heard of the method. However, I have not previously read it in full, and I do not remember anything about the authors. So this should still be a fully blind review. This paper introduces TAP -- a black-box method to develop attacks against LLMs. The attack process constructs a tree with a branching step based on refining a prompt using an LLM and a pruning step based on pruning attacks deemed off topic or assigned a low jailbreaking success score by another LLM. This method is thus able to search for attacks in a way that's automated and interpretable.

Strengths

S1: I think that the overall contribution is valuable to the field. TAP is kind of simple and arguably a glorified data-augmentation technique, but I think that TAP clearly belongs in the red-teaming toolbox. S2: One might criticize this paper for being an incremental improvement on PAIR, but I think that would be a pretty lazy criticism, and I don't buy that that would make this not valuable. I think that tree method is a good insight, and it doesn't have to be complicated to be useful.

Weaknesses

W1: I think that the biggest potential problem with this work is the jailbreak evaluation method. Recent work like [https://arxiv.org/abs/2402.10260](https://arxiv.org/abs/2402.10260) has shown that LLM autograders for jailbreak success tend to be pretty problematic. Meanwhile, I'm very skeptical of a supposed 88% success rate against GPT-40 in Table 8. Overall, I don't think that what is being evaluated is probably an ideal proxy for "safety". W2: The biggest limitation with TAP to me seems that its ability to help you find very novel jailbreaks seems very limited, and the initialization/prompting is doing a lot of work. I'd be appreciative of discussing this limitation or some more quantitative/qualitative analysis of it.

Questions

See weaknesses.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

See weaknesses.

Summary

This paper presents an automated method, Tree of Attacks with Pruning, for generating jailbreak prompts to exploit vulnerabilities in LLMs. TAP uses a tree-of-thought reasoning approach to iteratively refine and prune candidate prompts, significantly reducing the number of queries needed to successfully jailbreak LLMs like GPT-4.

Strengths

1. TAP's method of using tree-of-thought reasoning combined with pruning optimizes the search for effective jailbreak prompts, requiring fewer queries. 2. TAP demonstrates effectiveness across multiple LLMs and attack scenarios, including models with advanced protections like LlamaGuard. The evaluations show TAP successfully jailbreaks most state-of-the-art LLMs for over 70%.

Weaknesses

1. The format of this paper needs to be refined, such as Table 1,2,3. 2. The success of TAP heavily depends on the choice of evaluator LLM, with significantly reduced performance when using less powerful evaluators. 3. Much lower attack success rate on well aligned open-sourced model, Llama-2-7b-chat. Llama3 and gemma2 may be needed for evaluating the effectiveness of this method.

Questions

See weakness part.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The running time of this method is long.

Summary

This paper propose Tree of Attacks with Pruning (TAP) to jailbreak blackbox LLMs automatically. TAP has four steps, including branching, pruning, attack and assess, and pruning. TAP leverage an attacker (an LLM) to generate variations of the provided prompt, and the evaluator (another LLM) decide which variations to send to the target model. TAP is an advanced version of PAIR, reaching a higher success rate more effectively.

Strengths

1, TAP can reach a high success rate on GPT4, GPT4o and Gemini-Pro. 2, Compared to PAIR, TAP is more efficiency on most of the black box model, suggesting the effectiveness of the branching and pruning. 3, TAP shows a better transferability between models.

Weaknesses

1, As a method built on PAIR, with branching and pruning, the novelty of the method is limited. 2, In Fig.1, a typo "branching".

Questions

1, In table 7, the attacker’s system prompt is long and carefully designed. The prompts try to get the LLM to make up a story for making the harmful behavior more reasonable and align with human value. This well designed prompt may be a key to success in jailbreak. It would help if the paper includes replacing the prompt with other evaluator/attacker prompts and show some results to state the success rate comes mostly from pruning instead of prompt engineering. 2, In section 6 discussion and appendix A.2, TAP-No-Prune has a higher success rate than TAP Prune, because retains the w = 10 highest scoring prompts and deletes the rest. Does it suggest that the off-topic prompts get higher scores? 3, As the subtree in Figure 12-14, is the improvement of the attacker generated by evaluator? What's the prompts for generating these? And are the revised prompts share the same attacker's prompts as the original generation? 4, In the branching stage, does the length of the re-writing prompts affect the success rate? Perhaps adding some constraints on the generation length may also help the jailbreak success rate.

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

Please refer to questions.

Summary

This paper presents a novel jailbreaking attack based on the PAIR attack, enhanced with branching and pruning techniques. The attack generates multiple prompts through branching, then applies two pruning steps: removing off-topic prompts and eliminating prompts with low scores after testing them against the victim model. The method achieves a higher effectiveness with a lower number of queries than previous work. The study evaluates the attack's transfer performance, the effectiveness of the attack when LLaMA Guard is used as a defense, and includes an ablation study on the contribution of the different attack components.

Strengths

**Attack effectiveness** The attack is strong and uses less queries than previous work. The branching and pruning steps are effective in generating effective jailbreaks. **Thorough evaluation** The evaluation is thorough and includes transfer performance and the effectiveness of the attack against LLaMA Guard. Moreover, the ablation study provides insights into the contribution of the different attack components. **Clear writing** The paper is well-written and easy to follow (except for the redundancy mentioned in the Weaknesses below). **Thorough experimental details** The paper provides a lot of details about the experiments in the appendix, which is helpful for reproducibility.

Weaknesses

**Redundancy in the paper content**. I find subsection 1.2 with the basic description of TAP redundant and makes the Introduction section overly long. There is a big overlap with the content of section 3. I suggest merging the two sections. **Unclear calibration of the Judge model**. The judge model give scores that are between 1 and 10, however, in my experience while using LLMs, they are usually poorly calibrated when assigning scores. I have then two questions: 1) would it be feasible to look at how calibrated is the judge model compared to a human baseline? 2) how important is it that the model is actually calibrated when using it for the pruning step? It would be interesting to see how the pruning step would perform when selecting random prompts instead of those with the highest assigned score (except of course for the prompts that are considerered to be fully successful, with a score of 10). It seems to me that the score domain is excessively granular even for a human to assign scores, so using a coarser score scale might be more effective and meaningful. I understand that this is the same prompt as the one used in PAIR, but it seems to me that the score in TAP has a more crucial role than in PAIR.

Questions

- What average is used for the number of queries in table 2? Is it the arithmetic mean? Are there many outliers? What is the median? - See my questions about model calibration above.

Rating

7

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors discuss the limitations of their work.

Thanks, I think the paper should be accepted

Thanks, not too much more to say. I basically think that we are on the same page. I'll hold at a 7. But I would encourage the authors to mention explicitly the problems with autograders. I would also add that mentioning any differences in eval methodology that might make the numbers in your tables and the numbers in other papers' tables apples-to-oranges.