As large language models (LLMs) become increasingly prevalent across many real-world applications, understanding and enhancing their robustness to adversarial attacks is of paramount importance. Existing methods for identifying adversarial prompts tend to focus on specific domains, lack diversity, or require extensive human annotations. To address these limitations, we present Rainbow Teaming, a novel black-box approach for producing a diverse collection of adversarial prompts. Rainbow Teaming casts adversarial prompt generation as a quality-diversity problem and uses open-ended search to generate prompts that are both effective and diverse. Focusing on the safety domain, we use Rainbow Teaming to target various state-of-the-art LLMs, including the Llama 2 and Llama 3 models. Our approach reveals hundreds of effective adversarial prompts, with an attack success rate exceeding 90% across all tested models. Furthermore, we demonstrate that prompts generated by Rainbow Teaming are highly transferable and that fine-tuning models with synthetic data generated by our method significantly enhances their safety without sacrificing general performance or helpfulness. We additionally explore the versatility of Rainbow Teaming by applying it to question answering and cybersecurity, showcasing its potential to drive robust open-ended self-improvement in a wide range of applications.
Paper
Similar papers
Peer review
Summary
The paper addresses automatic red teaming of large language models through open-ended generation of jailbreaks. The key components of their methodology are 1) a categorization of different jailbreak categories to create a diverse archive of possible jailbreaks, 2) a strategy to evolve and mutate jailbreaks, 3) and a selection process to keep the jailbreaks with the highest quality.
Strengths
The paper presents a considerable contribution towards practical automatic redteaming of large language models and address multiple weaknesses of prior work (such as attack diversity). The authors provide an exhaustive empirical evaluation of the proposed method with extensive hyperparameter descriptions.
Weaknesses
W1: Despite the extensive lists of hyperparameters and explanations, the results are not realistically reproducible without the dataset of generated jailbreaks or the trained model. As adversarial robustness has shown to be brittle in the past, I strongly recommend any safety paper to make it as easy as possible to verify their defense. A considerable amount of powerful open-source large language models is available and there is no particular reason why releasing the trained robust model would provide any additional safety concern to the community. (Note that this is more of a personal concern and will not influence my score as I understand that it might have not been possible for the authors to release the model / code / data) W2: The robustness evaluation of the the model trained with rainbow teaming data is insufficient. I would argue that safety assessments can never be “fixed” and need to be adaptive and designed for the model at hand. The authors performed a train test split to evaluate the robustness of the model, which is non-adaptive. At least any evaluation with one of the many adversarial attacks proposed for LLMs in the last year would have put the robustness into better perspective. Most defenses proposed in the robustness domain have later been shown to be ineffective and “offline” adversarial training (generating the attacks prior to training) does not yield robustness for image models against stronger attacks in my own experiments. Thus, I am a bit sceptic if rainbow teaming actually improves worst-case adversarial robustness. W3: Evaluations are limited to the Llama series of models. Experiments on non-aligned models or models trained with less safety fine-tuning would have been interesting. E.g., "How does rainbow-teaming compare to standard model alignment?". (relatively minor concern) I am very likely to raise my score if the authors provide more results regarding the worst-case adversarial robustness of models trained with rainbow teaming or provide a good reason why this is not necessary / out of scope.
Questions
Could the authors conduct worst-case attacks against the model trained with rainbow teaming i.e., [1]? [1] Andriushchenko et al., “Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks”, 2024
Rating
7
Confidence
5
Soundness
3
Presentation
4
Contribution
4
Limitations
The authors provide an extensive list of limitations.
Concerns addressed
I thank the authors for their response. My concerns are addressed and I will increase my score accordingly. You are correct that the adaptive attack proposed by Andriushchenko is out of scope for the review and I apologize for bringing it up. Nevertheless, I think it would provide an interesting additional experiment for the final paper :-)
Thank you for a quick reply
We thank the reviewer for their blazing fast reply and for updating their score! While out of scope, we will consider the reviewer's suggestion to expand the set of evaluations to white box or token based attacks such as GCG [1] or Andriushchenko et al. [2] for the final paper. Regarding reproducibility, note that we will also be releasing a full archive of prompts produced by Rainbow Teaming in the final version of the paper. These prompts will provide further insight into the kind of jailbreaks discovered by Rainbow Teaming. They will also help reproducibility efforts, for instance by allowing researchers to warm-start a Rainbow Teaming run by initializing their archive with our prompts. Finally, if the reviewer has suggestions on how we can improve the paper and our score even further, we ask that they please let us know. [1] Zou et al. *Universal and Transferable Adversarial Attacks on Aligned Language Models*. 2023 [2] Andriushchenko et al., *Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks*, 2024
I highly appreciate the effort in making the paper reproducible, which I believe is the most relevant practical limitation of the current version (even if a lot of information is available, reproducing the results will be infeasible for most researchers). However, open-sourcing the dataset would enable future research on attacks and defenses. I will consider increasing my score and would like to give my opinion on some of the stated weaknesses of other reviewers: **Limited experiments / Bigger models.** The authors provide relevant experiments for safety fine-tuned and models trained without safety alignment and perform their attack on closed-source models. If we ask for more experiments as the bar to accept papers, we will not have any papers from academia on red-teaming LLMs/VLMs in the future. **Automated metrics.** An investigation of the ASR is provided in the paper. I think its unreasonable for every new red teaming paper to perform manual evaluations or do the same investigations as a lot of prior work on the human alignment of LLMs as a judge. **Novelty.** I believe this to be a very bad metric to judge a paper. "On Adaptive Attacks to Adversarial Example Defenses" was initially rejected for low novelty but had a big impact on the field and now has nearly 1000 citations. The kind of red-teaming studies performed in this paper are important to guide new approaches to defend and attack LLMs.
Message to Reviewer w9N7
We thank their reviewer for their continued engagement, and for their insightful and supportive comments. We are glad they appreciate our reproducibility efforts, including the dataset release. We also thank them for taking the time to read and comment on other reviews, therefore encouraging constructive discussion around our paper. We believe this sets a positive example for how to engage during the author-reviewer discussion period.
A note on visibility of Reviewer w9N7's comments
We noticed that reviewer w9N7’s past two comments, which are partly addressed to other reviewers, are only visible to authors and AC, but not to other reviewers. Could we ask reviewer w9N7 to edit their comments’ visibility so that it is visible to other reviewers (by ticking the necessary box below the comment area when editing a comment)?
Sorry for the oversight. I changed the visibility.
Summary
This paper introduces RAINBOW TEAMING, an approach for generating diverse adversarial prompts to test and improve the robustness of LLMs) The method uses quality-diversity search to produce a wide range of effective adversarial prompts across different categories. The authors demonstrate its effectiveness on state-of-the-art models like Llama 2 and Llama 3.
Strengths
RAINBOW TEAMING offers a new perspective on adversarial prompt generation by framing it as a quality-diversity problem.
Weaknesses
1. The study focuses primarily on Llama 2 and Llama 3 models, citing licensing constraints for not including other major models like GPT-4 or Claude. This focus limits the generalizability of the findings. It would have been more convincing to see results across a wider range of models from different providers, especially given the importance of the topic. 2. While the authors report high inter-evaluator agreement between GPT-4, Llama Guard, and humans on a small sample, the study relies heavily on automated metrics for evaluating the safety of responses. 3. While the paper mentions that fine-tuning with RAINBOW TEAMING-generated data improves model robustness, it lacks a detailed analysis of potential effects on the model's general performance or capabilities.
Questions
See the above.
Rating
4
Confidence
4
Soundness
2
Presentation
3
Contribution
2
Limitations
Yes
Summary
The authors present Rainbow Teaming, a structured approach to automated redteaming of large language models. Based on a user-specified set of strategies and risk categories, Rainbow Teaming uses a mutator LLM to rewrite several variations of existing prompts, then compares the resulting outputs from the target model with a judge LLM against the existing prompt. If a more effective prompt is found, it replaces the existing prompt in the "archive" of prompts found so far. The authors conduct experiments redteaming various open-weight models such as Llama and Mistral, showing that their method achieves a high success rate on various risk categories. They also explore the use of the generated prompts in supervised finetuning, showing that robustness to Rainbow Teaming can be improved by training against Rainbow Teaming.
Strengths
* Exposition of proposed method is very clear and effective * Lots of helpful figures and diagrams to illustrate the various components of the entire pipeline, such as the concept of the "archive" * Method appears quite effective at least against smaller, open-weight models
Weaknesses
* Contribution and novelty seems very marginal. The difference from methods such as PAIR and TAP appears to come down to 1) presenting the attack/mutate LLM with high level categories instead of specific behaviors and 2) specifying several concrete strategies instead of relying on the attack/mutate LLM to come up with them on the spot * Lack of comparisons to prior work. The authors offer various criticisms of PAIR, TAP, and the approaches studied by Perez et al. but do not show any evidence that their method outperforms or finds substantively different prompts from these approaches. Table 1 presents some results which I do not understand. Evaluations on common benchmarks such as AdvBench and HarmBench are missing. * Lack of experiments on bigger models. The authors only run Rainbow Teaming against 7B models. They claim they are unable to evaluate against more powerful models such as GPT and Claude because of legal constraints, but there are plenty of larger and more powerful models for which this is not a concern, such as Llama-3 70B and many others. * Fig 4 shows that Rainbow Teaming only improves performance by about 10% beyond the simple baseline of sampling lots of candidates from scratch, suggesting that the whole evolutionary framing of elites, mutations, etc is not so critical. * The experiments on improving robustness with training on Rainbow Teaming prompts in Sec. 5 are not convincing. The authors generate 15 sets of prompts targeted against Llama-2 7B, train on 12, and then show near-perfect performance on the held out 3 sets. How different are these 3 sets from the 12 training sets if they are generated with the same algorithm? But when running the Rainbow Teaming pipeline against the fine-tuned model, they still find a nearly 40% attack success rate which is not robust at all. And this is in spite of the fact that they do not appear to be using any holdout behaviors for validation.
Questions
* The authors criticize the methods in Perez et al. as costly, but use up to 2000 iterations and several thousand LLM calls in their proposed approach. How much more costly is Perez et al. than Rainbow Teaming? * Prior work has found that Llama-2 7b chat is significantly more robust if the default system message released with the model is used. Do the experiments here use that system message?
Rating
3
Confidence
4
Soundness
2
Presentation
4
Contribution
2
Limitations
Yes
A reply to Reviewer sYeN's full response
We thank the reviewer for further questions. Below, we address all the issues you raise in your response. ### **On No Stepping Stones Baseline** We agree that using stepping stones is a core aspect of our method, as well as the MAP-Elites style algorithms on which we base Rainbow Teaming. **However, our method performs significantly better than the No Stepping Stones baseline — their performance is by no means “largely the same”**. In the safety domain, Rainbow Teaming achieves 92% attack success rate (ASR) on the Llama 2-chat model, whereas the No Stepping Stones baseline achieves only 83% ASR. **This difference is significant.** In the domain of question answering, Rainbow Teaming outperforms No Stepping Stones baselines significantly on mean fitness of the archive, coverage, as well as the diversity of the archive (as evaluated using self-BLEU). See the table below for more information, which is the Table 3 from the paper. **This difference is, again, significant.** | Method | Mean Fitness ↑ | Coverage ↑ | Self-BLEU ↓ | | --- | --- | --- | --- | | Rainbow Teaming | 0.91 ± 0.01 | 0.97 ± 0.01 | 0.50 ± 0.02 | | Baseline (No Stepping Stones) | 0.79 ± 0.01 | 0.90 ± 0.01 | 0.60 ± 0.01 | If we proposed the baseline from Figure 4 as our main method, it would still have been novel, given the aforementioned algorithmic innovations. Also no prior method tackles adversarial prompt generation from a quality-diversity (QD) perspective or achieves an ASR close to what we did on various models. ### **Enhancing Adversarial Robustness** We demonstrated, convincingly, that performing supervised fine-tuning (SFT) on Rainbow Teaming-generated data significantly improves model’s safety robustness. Specifically, we showed that after performing additional SFT 1. The ASR of PAIR [1] **drops from 14% to 0%**, i.e., *PAIR can no longer jailbreak Llama 2 model, at all* (see the newly provided PDF attached to the response to all reviewers). 2. The ASR on previously unseen Rainbow Teaming generated archive **drops from 92% to 0.03%**, i.e., *previous jailbreaks created by our method don’t work, at all* (see Table 2). 3. The ASR of re-applying Rainbow Teaming from scratch results in a **drops from 92% to 39%**, *i.e., our method is struggling to jailbreak this new version of Llama 2* (see Figure 5)*.* 4. The safety reward model scores **increase from 0.883 to 0.897,** i.e., *even on unrelated Anthropic Harmless dataset the Llama 2-chat mode becomes much safer* (see Table 2). We strongly believe that the results above are more than conclusive. ### **Stronger attacks against the adversarial fine-tuned model** > *PAIR is a very weak attack, as evidenced by it's paltry 14% ASR on the not-yet-fine-tuned model….* the authors need to evaluate on stronger attacks. > There are no stronger black-box prompt-based attack in the literature than PAIR [1]. If the reviewer wants us to use more powerful jailbreaking techniques, they should clearly state which ones. ### **Llama-2 chat being non-robust** > *Llama-2 chat is a very non-robust model* > This statement is unfounded. Every prior work which evaluated on Llama-2 chat struggled with it in comparison with other methods. For instance, the original PAIR paper achieves 88% against Vicuna, 48% against GPT-4, but only **4% against Llama 2-chat** [1]. Similar differences can be seen in other works, such as [2] or [3]. Unless the reviewer can provide evidence to the contrary, we are convinced that there are no open-sourced models that are more robust than the Llama 2 and 3 series. ### **Attacking the fine-tuned model with Rainbow Teaming** > *the authors need to run their own attack on their fine-tuned model.* > **This is a key result of our original submission.** We present it in Figure 5 with the colorful rainbow curve, given additional results in Figure 13, and mention it in the conclusion. Perhaps the reviewer missed Figure 5, or they mean something else and wish to clarify? ### **Conclusion** We hope that our detailed response addresses the reservations you have. If you have further questions or concerns, please let us know. If not, we would appreciate if the reviewer could adjust their scores to reflect this. [1] Chao et al, Jailbreaking Black Box Large Language Models in Twenty Queries, 2023 [2] Paulus et al, AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs, 2024 [3] Zou et al. Universal and Transferable Adversarial Attacks on Aligned Language Models. 2023
The authors have not addressed my main concerns so I will just summarize them here again for the other reviewers and meta-reviewers: 1. The presented method is not considerably stronger than prior attacks [1, 2], and the proposed understanding of how it works is not convincing, as shown by the basically unchanged performance of their method when the iterative "evolution" component is ablated. Llama-2 7b chat is known to be a _weak_ model. Multiple teams of researchers have found simple methods that reliably achieve _100%_ ASR. Thus we know that the set of strings which can successfully trigger harmful behaviors is very, very large. In context, a 90% or 80% ASR after an arbitrary cutoff of 2000 steps is simply not meaningful when the ASR curve for the baseline method in Figure 4 is continuing to improve while the proposed method has plateaued. 2. Robustness evaluations of the fine-tuned model are unsound and incomplete. The authors lead with a bolded claim that fine-tuning reduces ASR from 92% to 0.3% but this is a totally unconventional evaluation which reuses test cases optimized against a completely different model. They also re-run their attack from scratch on the fine-tuned model, but without holding out any behaviors and bury this result much deeper in the text. The only other attack evaluated against the fine-tuned model is PAIR, which is a *weak* attack whose ASR is just not informative of model robustness. There are a wide variety of popular open-source attacks (from papers with many dozens of citations) the authors could have easily run against their new model because it is literally using the Llama-2 7b architecture + config which everyone evaluates against: https://github.com/llm-attacks/llm-attacks, https://github.com/RICommunity/TAP, https://github.com/centerforaisafety/HarmBench. The authors chose to ignore these works in their evaluation. I will maintain my score.
Further clarifications to Reviewer sYeN comments
### **Comparison with No Stepping Stones Baseline** > basically unchanged performance of their method when the iterative "evolution" component is ablated > The performance between our method and the baselines is **NOT unchanged**. We have highlighted their difference, several times. But it seems the reviewer chooses to ignore this. > Llama-2 7b chat is known to be a *weak* model. Multiple teams of researchers have found simple methods that reliably achieve *100%* ASR. > This is factually incorrect. Llama 2 and Llama 3 chat variants are arguably the safest open-source models. Even on papers the reviewer cites themselves [1-3], Llama models are still the hardest to jailbreak (see below for more information on this). Regardless, the reviewer has repeated their unsubstantiated claim on this topic. > In context, a 90% or 80% ASR after an arbitrary cutoff of 2000 steps is simply not meaningful when the ASR curve for the baseline method in Figure 4 is continuing to improve while the proposed method has plateaued. > Our method has been significantly better than the Baseline from step 0 to step 2000. Even if, hypothetically, they both achieve the same ASR at step 4000, our method would still be very clearly outperforming this baseline with a significant margin due to its sample efficiency throughout the entire search process. Also, we invented this baseline ourselves and it has substantial algorithmic complexity. The fact that it also works quite well is not a valid criticism to our main approach. We have reiterated these points to reviewer several times. We are perplexed by the reviewer’s continued refusal to engage with our arguments and by their dismissal of numerical results in Figure 4, which are clearly statistically significant. ### **Evaluations of Robustness** > *totally unconventional evaluation which reuses test cases optimized against a completely different model* > We take prompts optimized against Llama 2-chat and apply them to Llama 2-chat + Adversarial Fine-Tuning. Showing that the fine-tuned model is robust to attacks that jailbroke it before SFT is not an “unconventional evaluation”. It’s simply a matter of splitting data in a train and test set and showing improved test set performance post training (i.e. fine-tuning). It is the lowest bar for improved robustness and, had we not ran this evaluation, reviewers would certainly have requested it, and justly so. > *They also re-run their attack from scratch on the fine-tuned model, but without holding out any behaviors and bury this result much deeper in the text.* > This result is anything but buried, and we find such claims from the reviewer concerning. We dedicate Figure 5 and a full paragraph (Lines 253-259) concluding the section titled “*Enhancing Robustness with Synthetic Data*” to this result. Not only did we use a rainbow-colored curve in the plot to attract attention, we also literally bolded the relevant result information in the text on line 255. > *The only other attack evaluated against the fine-tuned model is PAIR, which is a weak attack whose ASR is just not informative of model robustness.* > The reviewer claimed the above and proposed GCG [1], TAP [2] and HarmBench [3]. These are inadequate: 1. GCG is a white-box token-based attack, whereas we study black-box prompt-based attacks. Please refer to Related Works (Section 7) and to Appendix D.1 - Token-Level Attacks for a discussion on this topic. 2. TAP has a self-reported ASR of 4% against Llama 2-chat 7B (See Table 1 of Mehrotra et al. [2]). Given the reviewer considered the 14% ASR of our implementation of PAIR against the same Llama 2-Chat model as “paltry”, we doubt they would have been satisfied with TAP. 3. HarmBench is an evaluation framework, not an attack method. Also, as stated in our original rebuttal, prompts from HarmBench are included in JailbreakBench, which is what we use when attacking models with PAIR in Table 2 Furthermore, we would like to remind the reviewer that our paper is not about defending LLMs. It is about discovering vulnerabilities. We include a section on enhancing robustness as a first demonstration of the utility of Rainbow Teaming datasets but, as stated in the paper, “*we leave the investigation of adversarial fine-tuning strategies to future work*”. [1] Zou et al. *Universal and Transferable Adversarial Attacks on Aligned Language Models.* 2023 [2] Mehortra et al. *Tree of Attacks: Jailbreaking Black-Box LLMs Automatically. 2023* [3] Mazeika et al., *HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal.* 2024 [4] Chao et al. *JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models*, 2024.
Summary
The paper proposes a novel method Rainbow Teaming for the automatic generation of diverse adversarial prompts aimed at large language models (LLMs). The goal is to identify and enhance the robustness of LLMs to various user inputs, which is crucial as these models are increasingly used in safety-critical environments.
Strengths
1. The proposed method holds significant importance in the current AI red teaming study for large language models (LLMs). 2. The proposed automatic method is straightforward to follow and appears to be effective on different open-source models.
Weaknesses
Lack of baseline comparisons in the safety evaluation for LLMs.
Questions
There are a few details that I'm not certain about: 1. What precisely does the Mutation Operator do? Does it mean using the LLM to generate the necessary risk category directly or in a few-shot fashion? 2. How is the Mutator LLM trained? What are the differences compared to a normal LLM?
Rating
4
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
I'm not sure if the authors plan to release their code or model checkpoint to facilitate further advancements in this field. Automatic red teaming is indeed crucial, but some methods can be challenging to reproduce since they often involve large amount of engineering work, which somehow is more likely an engineering work rather than a research study. Furthermore, while this paper primarily focuses on diverse risks, it would be beneficial to include a comparison with existing red teaming approaches, as they can also be utilized to evaluate and enhance the safety of LLMs.
Thank you for acknowledging
Thank you for acknowledging our rebuttal. Please let us know if you have remaining questions or concerns. If not, we ask the reviewer to please update their score to reflect this. Note that after further consideration, we will also include a full archive of prompts produced by Rainbow Teaming in the final submission, to help guide reproduction efforts.
A message to the reviewer
This is a gentle reminder that **tomorrow, August 13**, is the final day of the Reviewer-Author discussion period. We have carefully addressed all the questions and concerns the reviewer raised, including on the comparisons with baselines and the Mutator LLM. Additionally, we have provided a joint response to all reviewers, highlighting novel experimental results, such as transfer performance to GPT-4o and comparisons with new baselines. We will also include a dataset of adversarial prompts in the final version of the paper. If you have any further questions or suggestions, please let us know. **If our responses have successfully addressed your concerns, we kindly ask you to consider updating your score accordingly.** Thank you very much.
I do agree with the excellent experimental results in this paper, particularily on commercial models. However, as I have mentioned, I still have concerns on the open-source availability of such attack papers. In my opinion, due to the strong capabilities of LLMs and their random responses, jailbreaking LLMs works are much more complex than those adversarial attack works before. Therefore, it is crucial to ensure that the code and experiments are reproducible and as open source as possible. This would establish a more solid baseline and add formality to this field. Taking all of this into consideration, I will maintain my score as borderline reject this time. I appreciate the authors' contribution and effort in this paper and during this rebuttal.
Response to Reviewer hknM
We are glad we could address all of the concerns of reviewer hknM and that their only remaining issue with the work is that it's not open-source yet. Unfortunately, making the code open-source is __beyond our control at this time__. To compensate for this, we have made every effort to provide very thorough details of our approach in the paper, including an in-depth description of the algorithm, a thorough pseudocode, and the full set of hyperparameters that were used. We believe these resources should make it straightforward for others to reimplement Rainbow Teaming.
Thanks for the response. And I will consider my score.
A message to the reviewer sYeN
This is a gentle reminder that **tomorrow, August 13**, is the final day of the Reviewer-Author discussion period. We have carefully addressed all the questions and concerns the reviewer raised, including on the comparisons with baselines and experiments with bigger models and various system prompts. Additionally, we have provided a joint response to all reviewers, highlighting novel experimental results, such as transfer performance to GPT-4o and comparisons with new baselines. We will also include a dataset of adversarial prompts in the final version of the paper. If you have any further questions or suggestions, please let us know. **If our responses have successfully addressed your questions and concerns, we kindly ask you to consider updating your score accordingly.** Thank you very much.
A message to reviewer DdX9
This is a gentle reminder that **tomorrow, August 13**, is the final day of the Reviewer-Author discussion period. We have carefully addressed all the questions and concerns the reviewer raised, including on the automatic metrics for evaluating ASR and enhancing adversarial robustness with Rainbow Teaming-generated data. Additionally, we have provided a joint response to all reviewers, highlighting novel experimental results, such as transfer performance to GPT-4o and comparisons with new baselines. We will also include a dataset of adversarial prompts in the final version of the paper. We also note that **Reviewer w9N7** has also, kindly, provided their supportive opinion on the usage of automated metrics and novelty of the method. Please see their comment below. If you have any further questions or suggestions, please let us know. **If our responses have successfully addressed your questions and concerns, we kindly ask you to consider updating your score accordingly.** Thank you very much.
I will respond to the other points, but first to clarify on the stepping stones baselines: I'm going off your description in lines 212-214: "ignores past solutions in the archive and generates new prompts on the risk category, before applying the attack style mutation". Looking at the pseudocode in Alg 2, it sounds like the change being made is that x is always set to a randomly generated seed prompt in each iteration, and that sampling from the archive (x ~ G) no longer takes place. Is this correct?
Responce to Reviewer sYeN's comment
We thank the reviewer for their question. You are correct in your understanding. The No Stepping Stones baseline does not sample from the archive; that is, the step x ~ G is omitted. Instead, at each iteration, a new seed prompt is generated from scratch in a zero-shot manner based on a given Risk Category. We then apply a single mutation to it according to a given Attack Style. All other aspects between this baseline and our main method remain identical. Additionally, we've included further baseline results in the updated PDF shared with all reviewers. If you have any further questions or concerns, please let us know.
Full response
I don't believe I have misunderstood your method or experiments. "Stepping stones" is a core part of the method. Without it, the evolutionary framing doesn't make much sense since the method then boils down to rejection sampling from the 1-step mutator distribution until better candidates are found. And yet without stepping stones performance is largely the same, albeit optimization may be a bit less efficient. My main concern, which remains unaddressed by the response, is that the experiments on improving robustness are weak and unconvincing. Even though the proposed redteaming method is not the most novel in its approach-- prior work has examined diversity constraints such as the Trojan Detection Challenge at NeurIPS 2023 (https://trojandetection.ai/tracks, as well as systematic, multi-strategy paraphrasing (https://chats-lab.github.io/persuasive_jailbreaker/)-- there would still be a lot of value in developing a method that can be used for adversarial training. But the authors have no demonstrated this convincingly. Evaluating the fine-tuned model on prompts optimized against the prior, not-yet-fine-tuned model doesn't make any sense to me. The issue here isn't attack adaptivity, it's that the attack is not even targeted to the model under evaluation. Echoing Reviewer DdX9, I agree that the authors need to evaluate on stronger attacks. PAIR is a very weak attack, as evidenced by it's paltry 14% ASR on the not-yet-fine-tuned model (Llama-2 chat) which itself is a very non-robust model. At the very least, the authors need to run _their own attack_ on their fine-tuned model.
Summary
I'd like to thank the authors for submitting their work for review. I found the work insightful, inspiring, and high-quality. In short, the work has been a pleasure to review as a last-minute reviewer. The manuscript's primary contributions include: - **Rainbow Teaming Method.** A new methodology for automatically generating adversarial prompts through the lens of a quality-diversity problem. - **Demonstrative Evaluation for Safety.** A demonstrative evaluation of the Rainbow Teaming methodology's utility application to the task of identifying prompt vulnerabilities that exist in a series of generative models. This demonstration also validates the underlying components of the Rainbow Teaming approach (e.g., the choice and design of the Preference Model). - **Fine-Tuning Evaluation for Safety.** A fine-tuning experiment that illustrates how fine-tuning Llama-2-chat 7B on a dataset of 1,500 synthetically generated adversarial prompts with SFT reduced the attack success rate from 92% / 95% to 0.3% / 0.7%. - **Post-SFT Evaluation for Safety.** The authors re-apply Rainbow Teaming to the fine-tuned model produced from the Fine-Tuning Evaluation and report that the model is "substantially more robust to our approach, with a final ASR of 39% (down from 92%". - **Non-Safety Evaluations**. The authors contribute two additional experiments that illustrate the method's efficacy for the Question-Answering and Cybersecurity settings, each of which contribute a set of concise, abbreviated findings in their own right. The manuscript has several other minor contributions that aren't explicitly referenced as contributions, but should not go unnoticed (e.g., the extended taxonomy of safety risk categories that was previously defined by Inan et al.)
Strengths
Generally speaking, I find the work to be strong in its contribution, and I have no issue in acknowledging the paper's strengths -- because there are many! ### 1. Originality * Rainbow Teaming can be categorized as a synthetic data generation method for adversarial settings. Synthetic data generation methods that are similar in nature (e.g. PAIR, MAP-Elites) are recognized by the authors. * Despite bearing similarity in the fundamental approach, there are certainly aspects of originality that enable the method to distinguish itself from those that come before it. * Irrelevant of the methodology's originality, it can be argued that aspects of the evaluations are themselves original. ### 2.Quality * I view the quality of the work is high, and the conducted experiments sufficiently support the recognized the work as such. * The manuscript experiments related to safety progressively build on one another, providing incremental validation at for each of the steps that educated readers might expect to see when replicating the methodology on their own. The experiments use appropriate metrics and are accompanied with conclusive statements that are, for the most part, reasonable and believable. ### 3. Clarity * Given page limit requirements, I find the paper's presentation to be exceptional. The writing is crisp, clear, and to the point. * The authors have given clear time and attention to ensuring their work is digestible to readers. * The figures are also well-designed and make it quite easy to understand how the Rainbow Teaming aims to provide a more holistic evaluation of safety. One could argue that Figure 1's visual representation may be appropriate for adoption as model providers continue to champion safety as an area of investment. ### 4. Significance * I view the work as an amalgamation of several existing concepts that, when stitched together as a collective, can be viewed as a significant contribution. * Rainbow Teaming's applicability to safety is clear and obvious, and the secondary evaluations on Question-Answering and Cybersecurity elevate the work's significance. * It's easy to imagine a generalization of the Rainbow Teaming methodology being applied to settings that aren't adversarial in their nature. The Appendix should be recognized as a strength in itself. The quality of depth and thoroughness is appreciated, even if some sections may not be as open as I'd like.
Weaknesses
The work has several weaknesses that should be taken seriously, but not viewed as disqualifying. I view each of the following weaknesses as nothing more than "expected". The weaknesses are as follows: 1. **Longitudinal Practicality.** The authors make a number of claims about the Rainbow Teaming method's ability to improve the robustness of generative models. While this is clearly demonstrated in the manuscript's family of experiments, the claim is weakened by the notion that the experiments do not provide information about how the Rainbow Teaming method may operate over time (i.e., in which adversarial methods evolve in new and unexpected ways). 2. **Attack Styles and Risk Categories.** The paper's contributions are bound by a static set of attack styles and risk categories. It remains unclear if the methodology would perform similarly with other styles or categories. 3. **Minor Weaknesses.** There are two minor weaknesses: - **Model Choice.** Conducted experiments are performed with models that are now viewed as potentially being dated (i.e., and are no longer state-of-the-art). This weakness is stated out of recognition that the model choice itself is a *potential* weakness. Regardless of whether this be recognized more formally as a weakness, I strongly believe that reviewers refrain from scrutinizing the choice of models as the work simply uses them as a vehicle for demonstrating their methodology. - **Diversity Metrics.** Diversity is primarily measured via BLEU, which one flavor of measurable diversity. Common practice is increasingly gravitating toward the measurement of multiple metrics that are reported as a collective, e.g. https://arxiv.org/html/2403.00553v1.
Questions
1. Can you please clarify the significance of the initial adversarial prompt sampled from the archive? Specifically, describe precisely what might happen in the Rainbow Teaming method if the sampled adversarial prompt is no longer effective in its attack. 2. What was the motivation behind choosing Question-Answering and Cybersecurity as two secondary areas of evaluation? 3. In Appendix A, the authors report that "Rainbow teaming requires extensive computational resources". While I understand the MAP-Elites approach may be computationally intense, it isn't exactly clear to me why the method itself is any more computationally intensive than preexisting methods. 4. In Appendx I4, the authors state that they've opted to refrain from sharing the prompts that facilitate mutation. To some degree, this seems nonsensical and misaligned to the broader goals of democratizing AI safety research in the community. Can the authors please provide a deeper explanation and justification for refraining to share the prompts?
Rating
8
Confidence
4
Soundness
3
Presentation
4
Contribution
3
Limitations
The authors acknowledge several key limitations of the Rainbow Teaming approach in Appendix A. Generally speaking, they are sufficient, but are not comprehensive or clear as I note in describing weaknesses and questions.
Response by Authors (1/2)
We are very grateful to Reviewer E172 for their extremely detailed review. We are pleased that they found our contributions significant and its presentation exceptional. It is also great that the reviewer appreciated the thoroughness of our empirical results. We address your questions and concerns below. Please let us know if you have any recommendations for further improving the paper and strengthening your support. ### **Longitudinal Practicality** As mentioned in App. A, we agree that a main limiting factor of our method is the fixed nature of the archive. We shared a new result in the Response to all Reviewers showing robustness to out-of-distribution attacks from another method (PAIR), but indeed future jailbreak methods might nullify the robustness imparted by the current round of fine-tuning. However, the versatility of our method means that developers can quickly react to newly discovered attacks by simply expanding the list of Attack Styles and rerunning Rainbow Teaming to get new fine-tuning data. ### **Static Set of Attack Styles and Risk Categories** We agree that having a dynamic set of attacks and risk categories would be a valuable addition to the paper. Rainbow Teaming can be extended so that LLMs define archive categories automatically, either before the search process, or more interestingly, throughout the search process as the archive gets filled over time. We have decided to leave this direction for future work, as we believe it warrants more extensive investigation than can be accommodated within this manuscript. Also, two results demonstrate the generality of our method, even to categories beyond those in the paper. The first is that our method remained effective when targeting JailbreakBench behaviors instead of the original 10 risk categories, despite the fact that it is not designed to elicit very specific harmful behaviors. The second is that it was equally effective across all safety, cybersec and Q&A, which is a much broader change than just providing new categories. ### **Minor Weaknesses** - **Diversity Metrics.** - We thank the reviewer for providing a reference for additional metrics of diversity. We agree that more metrics of diversity would strengthen the paper and we can commit to include more in the final version of the paper. - Additionally, we use BERTScore in the ablation study for mutation filtering (see Table 5). - **Model Choice.** - We made every effort to include the most capable, recent and safest open models. For example, Llama 3 was released on April 18, less than a month before the deadline. We also show transfer results to GPT-4o in our response to all reviewers. - We strongly agree with the reviewer that here, in particular, the set of 8 models that we use are mainly vehicles for demonstrating our methodology.
Response by Authors (2/2)
### **Questions** > Q1. significance of the initial adversarial prompt When adding a prompt to the archive, we cache its elicited Target response. When sampling a parent prompt in the Selection phase (Figure 2), the cached response is ignored, and Mutation uses only the prompt itself and the candidate prompt descriptor. During Evaluation, the Judge compares the (newly generated) response elicited by the candidate prompt to the cached response to the prompt already in the archive at that position. We never reprompt the Target on prompts already in the archive. The Update happens if the response to the candidate prompt is judged more harmful than the cached response in at least 3 out of 4 comparisons by the Judge. This process is identical for the initial prompts that are generated at random and placed in the archive before the MAP-Elites search process and for those observed later at any point. > Q2. motivation behind QA and Cybersec We aim to include other areas that are substantially different than safety. Question Answering is orthogonal and complementary to safety. Specifically, we aim to diagnose features of the target LLM typically acquired during pre-training (general knowledge), whereas safety and alignment are performed during post-training stage either through SFT or RLHF. The cybersecurity domain’s main difference from that of safety is that the response from the LLM is around unsafe code rather than text only. It also featured established open-sourced evaluators (CyberSecEval) and risk categories (MITRE) we could rely on to assess the performance of our method. > Q3. On computational requirements The computational intensiveness is indeed due to MAP-Elites component of the approach. There are no additional bottlenecks. We stressed that Rainbow Teaming requires extensive computation resources to distance it from a line of work that aims to find jailbreaks with minimal compute or time. While that line of work is valuable on its own right, here we concentrate on generating a large collection of diverse and high-quality adversarial prompts, which is simply a different use case. > Q4. Mutation prompts *It was decided* that, while the Judge and Evaluation prompts are essential to reproducibility and are mostly positive in nature since they focus on harm appraisal, the Mutation prompts provide a non-negligible risk to the community. This is because they turn the LLM into an attack generating program, which carries a potential for misuse. As a result, and because we are fully aligned with the goal of democratizing AI safety research, the authors opted to describe the mutator in detail instead. We note that there is no “secret sauce” to the prompt — it is merely a few shot prompt asking the model to change the parent prompt to match the prescribed Risk Category or Attack Style. Any reimplementation effort is also likely to use a newer model than we did (a variant of Llama 2-chat 70B), which would require rewriting the prompt for maximum performance. We once again thank the reviewer for their extensive review under such a tight deadline. We wish we had more time to engage with them, but hope we provided a satisfactory answer to their questions.
Decision
Accept (poster)