We introduce WildTeaming, an automatic LLM safety red-teaming framework that mines in-the-wild user-chatbot interactions to discover 5.7K unique clusters of novel jailbreak tactics, and then composes multiple tactics for systematic exploration of novel jailbreaks. Compared to prior work that performed red-teaming via recruited human workers, gradient-based optimization, or iterative revision with LLMs, our work investigates jailbreaks from chatbot users who were not specifically instructed to break the system. WildTeaming reveals previously unidentified vulnerabilities of frontier LLMs, resulting in up to 4.6x more diverse and successful adversarial attacks compared to state-of-the-art jailbreak methods. While many datasets exist for jailbreak evaluation, very few open-source datasets exist for jailbreak training, as safety training data has been closed even when model weights are open. With WildTeaming we create WildJailbreak, a large-scale open-source synthetic safety dataset with 262K vanilla (direct request) and adversarial (complex jailbreak) prompt-response pairs. To mitigate exaggerated safety behaviors, WildJailbreak provides two contrastive types of queries: 1) harmful queries (vanilla & adversarial) and 2) benign queries that resemble harmful queries in form but contain no harm. As WildJailbreak considerably upgrades the quality and scale of existing safety resources, it uniquely enables us to examine the scaling effects of data and the interplay of data properties and model capabilities during safety training. Through extensive experiments, we identify the training properties that enable an ideal balance of safety behaviors: appropriate safeguarding without over-refusal, effective handling of vanilla and adversarial queries, and minimal, if any, decrease in general capabilities. All components of WildJailbeak contribute to achieving balanced safety behaviors of models.
Paper
Similar papers
Peer review
Summary
This paper introduces an automatic red-teaming framework that mines in-the-wild user chat logs and discovers various unique clusters of novel jailbreak tactics, which results in more diverse and successful adversarial attacks compared to state-of-the-art jailbreaking methods. Based on this framework, this paper creates a large-scale open-source synthetic safety dataset with much vanilla and adversarial prompt-response data pairs. This dataset enables the machine learning communities to examine the scaling effects of data and the interplay of data properties.
Strengths
1. This paper conducted a comprehensive study on the current in-the-wild jailbreak attacks. The two challenges mentioned in the Introduction are well addressed in this paper. 2. The two-stage framework is reasonable. The first step involves collecting the real-world user conversation logs and deconstructing them, while the second step composes the jailbreak tactics and construct them with vanilla harmful queries. 3. The experiment are thorough and demonstrate the efficiency of the framework. The experimental results in Section 5.2 validate the utility of the benchmark in safety training.
Weaknesses
1. Some method details missed. This paper lacks details on how the tactics are selected for constructing the final jailbreak prompt from an initial harmful prompt, as mentioned in lines 126-128. Table 1 lists many tactics, but it is unclear how these tactics are chosen. 2. Some metrics are vague: The process for constructing jailbreak prompts, as described in Sections 2 and 3.1, is fixed. However, the metric “Query” in Table 2 changes with the target model. This discrepancy needs explanation. Are there additional details about the method not mentioned in the paper? 3. The evaluation methods is not rationale enough: In Table 2, diversity is evaluated using sentence embedding similarity. This straightforward method might not work in all cases. For example, if the similarity between A and B, B and C, C and A is 0.8, 0.2, and 0.3, respectively, the average similarity for A, B, C would be 0.55, 0.5, and 0.25, respectively, categorizing them into three parts since they are all lower than 0.75. However, they should be categorized into two parts {A+B, C}. Additional details and justification for the similarity-based metric are needed to prove its rationality.
Questions
1. See (1) in weaknesses. How these tactics are selected? Are all these tactics available for almost all the harmful prompts? If not, how should you ensure the semantic rationality of the constructed jailbreak prompt? 2. See (2)&(3) in weaknesses. Could you give more details about each metric and the similarity-based metric and prove their rationality?
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
4
Limitations
Authors have not demonstrated the limitations specifically. Limitations from this reviewer can be seen in Weaknesses and Questions.
Thank you for taking the time to review our rebuttal and for recognizing that it helps address previous concerns!
Dear Reviewer `wDdc`, Thank you for taking the time to review our rebuttal and for recognizing that it helps address previous concerns! If our response has satisfactorily resolved the questions raised, we would greatly appreciate your consideration of adjusting the score, if deemed appropriate. Thank you once again for your time and effort. Sincerely, Authors of Paper 12439
Summary
The paper introduces WILDTEAMING, an automatic red-teaming framework that mines user-chatbot interactions to discover novel jailbreak tactics against large language models (LLMs). It uncovers vulnerabilities of frontier LLMs, leading to more diverse and successful adversarial attacks. The authors also created WILDJAILBREAK, a large-scale open-source synthetic safety dataset comprising 262K prompt-response pairs to enhance safety training. The paper demonstrates the effects of data properties and model capabilities during safety training, identifies training properties for balanced safety behaviors, and shows that WILDJAILBREAK improves model safety without diminishing general capabilities. The authors emphasize the need for shared norms and technical standards in LLM safety and call for further research into safety training methods and comprehensive safety evaluation.
Strengths
Originality: - The WILDTEAMING approach of mining jailbreak tactics from real-world user interactions and composing them into diverse attacks is novel and creative. - The creation of the WILDJAILBREAK dataset with four contrastive components (vanilla/adversarial harmful/benign queries) is an original contribution. - The paper introduces new diversity metrics for evaluating jailbreaking methods. Quality: - The methodology appears rigorous, with careful mining of jailbreak tactics, automatic scaling using GPT-4, and extensive experiments. - Comparisons to multiple strong baselines demonstrate the effectiveness of the approach. - The analysis of scaling effects and interplay between data properties and model capabilities is thorough. Clarity: - The paper is generally well-structured and clearly written. - Key concepts and methods are explained in detail. - Figures and tables effectively illustrate the approach and results. Significance: - The WILDTEAMING framework provides a practical way to identify diverse vulnerabilities in language models at scale. - The WILDJAILBREAK dataset is a valuable open resource for safety training. - The insights on scaling effects and balanced safety behaviors are important for developing safer AI systems. - The work addresses critical challenges in AI safety and ethics. Overall, this paper makes substantial original contributions, demonstrates high-quality research, clearly communicates the methods and findings, and has significant implications for improving the safety and robustness of large language models. The combination of novel techniques, extensive experimentation, and practical resources make this a strong and impactful work in the field of AI safety.
Weaknesses
1. Limited analysis of ethical implications: While the paper focuses on improving AI safety, it lacks a thorough discussion of the ethical implications of creating and releasing a large dataset of harmful queries and jailbreak tactics. The authors should address: - Potential misuse of WILDJAILBREAK by malicious actors - Safeguards implemented to prevent unintended harm - Ethical considerations in mining real-world user interactions for jailbreak tactics Suggestion: Include a dedicated section on ethical considerations, discussing the above points and outlining measures taken to ensure responsible use of the dataset and methods. 2. Insufficient comparison to human-generated jailbreaks: The paper compares WILDTEAMING primarily to automated methods like PAIR and AutoDAN. However, it lacks a direct comparison to human-generated jailbreaks, which could provide valuable insights into the effectiveness of the approach. Suggestion: Conduct a small-scale study comparing WILDTEAMING-generated jailbreaks to those created by human experts or crowdworkers. This would help contextualize the method's performance relative to human creativity. 3. Limited exploration of model-specific vulnerabilities: While WILDTEAMING is presented as a model-agnostic approach, it's possible that certain models may be more vulnerable to specific types of jailbreak tactics. The paper doesn't explore this aspect in depth. Suggestion: Analyze the effectiveness of different jailbreak tactic categories across various model architectures and sizes. This could reveal interesting patterns in model vulnerabilities and inform more targeted defense strategies. 4. Lack of long-term effectiveness assessment: The paper doesn't address how the effectiveness of WILDTEAMING-generated jailbreaks might change over time as models are updated or as the approach becomes widely known. Suggestion: Discuss the potential limitations of the approach in a "Limitations and Future Work" section. Consider proposing a longitudinal study to assess the long-term effectiveness of the generated jailbreaks. 5. Insufficient analysis of computational costs: While the paper mentions that WILDTEAMING has "low computational costs," it doesn't provide a detailed analysis of the computational requirements compared to other methods. Suggestion: Include a quantitative comparison of computational resources required for WILDTEAMING versus other jailbreaking methods. This would strengthen the claim of practicality for scaling up. 6. Limited discussion of failure cases: The paper focuses primarily on the successes of WILDTEAMING but doesn't provide an in-depth analysis of cases where it fails or performs suboptimally. Suggestion: Include a section analyzing failure cases, discussing the types of jailbreaks that WILDTEAMING struggles to generate or identify. This could provide valuable insights for further improvement of the method. 7. Lack of ablation studies on jailbreak tactic categories: While the paper presents an ablation study on pruning methods, it doesn't explore the relative importance of different categories of jailbreak tactics. Suggestion: Conduct ablation studies removing different categories of jailbreak tactics to understand their individual contributions to the overall effectiveness of WILDTEAMING.
Questions
1. Ethical considerations and safeguards: - What specific measures have been taken to prevent misuse of the WILDJAILBREAK dataset? - How do you address potential privacy concerns related to mining real-world user interactions? - Have you consulted with ethics experts or an ethics review board during this research? 2. Comparison to human-generated jailbreaks: - Have you considered conducting a comparative study between WILDTEAMING and human experts in generating jailbreaks? - If so, what challenges do you foresee in such a comparison? 3. Model-specific vulnerabilities: - Did you observe any significant differences in the effectiveness of certain jailbreak tactics across different model architectures or sizes? - Could you provide more detailed analysis on how WILDTEAMING performs across a wider range of models? 4. Long-term effectiveness: - How do you anticipate the effectiveness of WILDTEAMING-generated jailbreaks to change over time as models are updated? - Have you considered strategies for maintaining the relevance of the approach as AI systems evolve? 5. Computational efficiency: - Can you provide a more detailed comparison of the computational resources required for WILDTEAMING versus other jailbreaking methods? - How does the efficiency of WILDTEAMING scale with the size of the target model? 6. Analysis of failure cases: - Could you elaborate on specific types of jailbreaks or scenarios where WILDTEAMING performs suboptimally? - How might these failure cases inform future improvements to the method? 7. Jailbreak tactic categories: - Have you conducted ablation studies on different categories of jailbreak tactics? - If so, what insights did you gain about the relative importance of various tactic types? 8. Generalizability beyond chatbots: - How well do you expect WILDTEAMING to generalize to other types of language models or AI systems beyond chatbots? - Have you tested or considered applying the method to other domains? 9. Integration with existing safety measures: - How do you envision WILDTEAMING and WILDJAILBREAK being integrated into existing AI safety practices? - Have you explored combining your approach with other safety techniques? 10. Potential for defensive applications: - Could the WILDTEAMING framework be adapted for proactive defense against jailbreaks? - Have you considered using the method to automatically generate safety training data in real-time? 11. Scalability of the approach: - As language models continue to grow in size and capability, how do you expect the scalability and effectiveness of WILDTEAMING to be impacted? - Are there any foreseeable limitations to the approach as AI systems become more advanced?
Rating
5
Confidence
5
Soundness
3
Presentation
2
Contribution
3
Limitations
1. Ethical concerns: The creation and release of a large dataset of harmful queries and jailbreak tactics could potentially be misused by malicious actors. 2. Generalizability: The effectiveness of WILDTEAMING may be limited to the types of models and interactions present in the mined datasets (LMSYS-1M and WILDCHAT). 3. Temporal relevance: As AI models evolve, the jailbreak tactics discovered by WILDTEAMING may become less effective over time. 4. Dependency on existing language models: The COMPOSE stage relies on off-the-shelf LLMs like Mixtral-8×7B and GPT-4, which may introduce biases or limitations. 5. Lack of human baseline: The paper doesn't provide a direct comparison between WILDTEAMING and human-generated jailbreaks, which could offer valuable context. 6. Model-specific vulnerabilities: The approach may not fully capture vulnerabilities that are specific to certain model architectures or training paradigms. 7. Computational requirements: While mentioned as having "low computational costs," the exact resource requirements and scalability of WILDTEAMING are not thoroughly detailed. 8. Limited exploration of failure cases: The paper focuses more on successes and less on scenarios where WILDTEAMING is less effective. 9. Potential for adversarial adaptation: As jailbreaking methods become known, AI developers may adapt their safety measures, potentially reducing the long-term effectiveness of WILDTEAMING. 10. Privacy considerations: Mining real-world user interactions for jailbreak tactics may raise privacy concerns, even if the data is anonymized. 11. Scope limitation: The focus is primarily on text-based language models and may not generalize well to other types of AI systems. 12. Potential for over-optimization: There's a risk that models trained on WILDJAILBREAK might become overly cautious, potentially affecting their performance on benign tasks. 13. Dependency on existing safety classifiers: The method relies on existing tools like the OpenAI Moderation API and LLAMA-GUARD, which may have their own limitations or biases. 14. Limited cross-cultural analysis: The paper doesn't extensively explore how jailbreak tactics might vary across different languages or cultural contexts. 15. Potential for false positives: The method might incorrectly classify some benign queries as jailbreak attempts, leading to unnecessary restrictions.
Summary
This work proposes, for LLM safety, a new jailbreak attack method, WildTeaming, and a new safety alignment dataset, WildJailbreak. WildTeaming first manually identify seed jailbreak tactics from large-scale in-the-wild adversarial user query datasets and then compose jailbreak attack by rewriting a vanilla harmful query into an adversarial harmful query based on the multiple identified tactics using a LLM. WildJailbreak is then built based on WildTeaming with balanced distribution of vanilla and adversarial, harmful and benign, samples.
Strengths
1. The scale and the diversity of attack and safety alignment dataset are large. They can thus complement the existing works. 2. The effort of identifying comprehensive tactics for jailbreak is appreciated. 3. The idea of using a LLM to compose multiple jailbreak tactics into one jailbreak attack prompt is interesting. 4. The opensource of large-scale safety alignment dataset could contribute a lot to future safety alignment research.
Weaknesses
1. The discussion of related works is insufficient. Specifically, many jailbreak methods are only referred yet without any description at Line 301. There are many different categories of jailbreak attack methods. They should be disccused in more details. The same issue is also applied to the review of safety enhancement methods of LLMs. Besides, the paper can be enhanced by providing more detailed comparison with the closely related works. 2. The motivation of diverse jailbreak tactics is weak. Following the convention of adversarial ML, given a vanilla harmful query, the objective should be to find the most effective tactic to rewrite the query to trigger the harmful output, instead of to find many different tactics. Therefore, I am not convinced by the evaluation protocol of comparing different jailbreak methods in terms of diversity. Furthermore, it seems that increasing the diversity of jailbreak tactics seems to only result in a marginal improvement in ASR or even lower than PAIR in Tab. 2 for Vicuna-7B. 3. The evaluation and comparison of the proposed methods are insufficient. First, the proposed attack is not tested against a broad range of defenses, some alternatives including rephrasing [1] and prompt engineering [2]. Second, the proposed safety alignment dataset is not compared against other external methods. Safety-Tuned LLaMAs is a closely related work but not empirically compared at all. Therefore, I agree with the effectiveness of proposed methods but feel hard to compare them against the latest progress in the field.
Questions
1. One of the listed tactics in Tab. 1 (right) is "ignore prev. instructions". Is not it a method of prompt injection?
Rating
7
Confidence
3
Soundness
3
Presentation
2
Contribution
3
Limitations
The limitations and ethical considerations of the proposed method are discussed in section 7.
Thank you for taking the time to review our rebuttal! Follow-up Response (post 1)
We sincerely appreciate reviewer `ZrdC`'s follow-up questions and active engagement during the discussion! In this response, we address the additional inquiries and strive to fully resolve any remaining concerns. If our response has satisfactorily resolved the questions raised, we would greatly appreciate your consideration of adjusting the score, if deemed appropriate. We also welcome further discussion if the reviewer has additional points and are happy to incorporate any further suggestions into our final paper. > Full Draft of Updated Related Work - **Red-Teaming and Jailbreaking LLMs.** Early attempts at red-teaming and understanding LLM vulnerabilities have focused on hand-crafting prompts and analyzing model responses [2, 22, 54, 69]. However, manual methods had quickly become impractical due to their prohibitive costs and lack of scalability. Thus, automated red-teaming and jailbreaking methods are developed for more scalable audit of model vulnerabilities [56]. One genre of methods involves gradient optimization that requires back-propagating through model parameters [96, 27, 28, 67]. However, they are computationally expensive, cannot be applied to black-box models, and often result in gibberish texts. There are also generation-based approaches (most related to our work) which generate jailbreaking prompts directly or through iterative edits [8, 49, 43, 44, 56, 7, 52, 87, 40, 88, 90, 19]. Other jailbreaking works study attacks during decoding time (e.g., decoding configurations [32], logit manipulation [92]), in other modalities (e.g., vision-language [68, 85, 66], LLM agents [61]), under multilingual settings [20, 86, 60], in programming mode [41], or through multi-turn interactions [62, 83, 63]. However, most existing automatic red-teaming and jailbreak methods rarely result in large-scale training resources for model safety enhancement due to their limited coverage of attack strategies and risk types, slow speed, or closed-source access [65]. WildTeaming differs from previous works by efficiently composing *diverse* adversarial attacks utilizing real-world jailbreak tactics mined from in-the-wild user-chatbot interactions. WildTeaming allows scalable synthetic safety training data generation in addition to simply showing its attack efficacy. - **Safety Evaluation and Safety Training of LLMs.** Many red-teaming efforts on LLMs have been formalized as benchmarks for evaluating model vulnerabilities—these typically are composed of harmful prompts that models should refuse [6, 78, 77, 71, 51, 23, 75, 9]. Meanwhile, to mitigate the potential byproducts of safety training, other benchmarks measure exaggerated safety behavior on benign queries [64, 15]. While LLM safety evaluation has been an active area of research, studies and resources for safety training have been *limited*, especially in adversarial settings [22, 16, 84]. Most related to our work in this space are Safety-Tuned Llamas [3] and SafeRLHF [16], which primarily focus on *vanilla harmful* queries by releasing small-scale safety training datasets and large-scale pairwise preference datasets, respectively. WildTeaming distinguishes from these works by releasing higher quality (shown by our training ablation experiments) and larger scale sequential instruction-tuning data consisting of both *vanilla* and *adversarial* queries. WildJailbreak also uniquely contains large-scale contrastive *benign* queries, in addition to harmful ones, used for mitigating exaggerated safety behavior (i.e., over-refusal). Finally, synthetic data has been used for LLM safety [7, 58,15]. Most relevant to our work is Rainbow Teaming [65], which uses synthetic data to populate a grid of attack spaces based on the attack style and risk category. However, their data and code are not publicly available. Our work differs in automatically mining diverse human-devised jailbreak tactics rather than manually defining attack styles [65], creating a large-scale open safety training resource that supports extensive safety training experiments. *We have omitted the citations corresponding to each number entry to avoid cluttering the response. However, we are again happy to share them if this is helpful information for the reviewer.*
Follow-up Response (cont. post 2)
> Further Clarification on the Diversity Metrics We appreciate the reviewer's further comments on the diversity measure of attacks, which has helped us understand the source of our differing views. We believe this discrepancy arises from the distinct roles we attribute to automatic jailbreak and red-teaming methods. The reviewer assesses these methods from a *security attacking perspective*, where the focus is on successfully breaking a model, typically quantified by the success rate of a single attack. In this project, we view these methods from a *model developer's perspective*, treating them as scalable model auditing tools designed to automatically and comprehensively identify model vulnerabilities. For such auditing purposes, diversity is crucial because it allows the method to uncover a wide range of failures. This emphasis on diversity in red-teaming is also supported by previous work (e.g., Rainbow Teaming [1]), though their data and code are not publicly available for comparison. By defining diversity metrics in our study, we hope to encourage future methods to consider incorporating the practical aspect of better supporting model auditing. *We will clarify our motivation further in the final paper.* In response to the reviewer’s point regarding our method enhancing diversity in tactics for implementing the same type of harm, rather than exploring different harm categories, it is important from the model developer's standpoint to consider both dimensions: (1) harm categories (e.g., misinformation, cybersecurity) and (2) jailbreak tactics. Our objective is to identify as many instances of harmful model responses as possible, covering both diverse harm categories and various methods of eliciting harm within a single category. Thus, both dimensions are crucial for improving model safety, also noted by previous literature [1]. The standard evaluation setup for auto-jailbreak methods typically fixes the harm category dimension, focusing on applying jailbreak strategies to provoke a specific harmful response, without uncovering new types of harmful behaviors. Within this evaluation framework, our work contributes by employing diverse novel strategies to trigger the targeted harmful behavior, which may aid in developing robust defenses against such behavior despite the variety of elicitation techniques. Although we do not explicitly focus on diversifying harm categories during the attack experiments due to the constraints of the standard evaluation setup, we address this in the construction of the WildJailbreak dataset. This dataset includes 50K harmful queries across 4 high-level categories (e.g., malicious uses, harmful language, misinformation, privacy) and 12 subcategories, significantly expanding the resources available for defining diverse vanilla harmful behaviors (examples in Table 23 in the Appendix). [1] Rainbow teaming: Open-ended generation of diverse adversarial prompts, 2024.
Follow-up Response (cont. post 3)
> Further Comparison of Defense Baselines We again thank the reviewer for following up on the external defense baselines. Following the new pointers from the reviewer, we ran new evaluations to compare to the SmoothLLM and Self-Reminders defense baselines. | Method | ASR on the WildJailbreak Adversarial Evaluation Set (2K+ examples) $\downarrow$ | |------------------------------------|--------------------------------------------------------------------| | Tulu2 | 71.0 | | Tulu2 (+ trained on WildJailbreak) | 1.7 | | Tulu2 (+ Self-Reminders) | 41.9 | | Tulu2 (+ SmoothLLM) | 42.2 | Our results show that applying the two external defense methods to Tulu2 reduces the attack success rate (ASR) from 71% to around 42%. However, training Tulu2 on WildJailbreak decreases the ASR to just 1.7%, making it significantly more effective than Self-Reminders and SmoothLLM. Additionally, it’s worth noting that the SmoothLLM defense requires duplicating and perturbing each prompt many times (in our experiment, five), which substantially slows down model inference. We hope these experimental results further strengthen the reviewer's confidence in the effectiveness of our safety alignment dataset. Lastly, we would like to take a moment to clarify why we consider extrinsic defense comparisons tangential to achieving our primary research goal of training safer models in our initial rebuttal. While we agree that model defense requires systematic solutions at various intrinsic and extrinsic stages, our work focuses specifically on *training a model to be safer by developing an improved safety training resource*. Extrinsic defenses address challenges very different from making a model intrinsically robust. The main issue is that applying extrinsic defenses to mitigate real-world harms is impractical with the current usage of *open-source* language models. Users typically interact with these models by inputting queries directly into the model and receiving outputs, but there's no way to enforce the use of extrinsic defenses during this process—like prompt engineering (Self-Reminders), perturbation injections (SmoothLLM), or safety filters (LlamaGuard)—on open-source models. These methods rely on voluntary application, which malicious users would avoid when exploiting models for harmful purposes. As a result, extrinsic defenses cannot replace intrinsic safeguards within the models themselves, making developing safety training resources, like WildJailbreak, essential and an independent challenge. *We will add the results of the new defense baselines and the discussion of internal and external defense to the final paper to further ground our contributions. We greatly appreciate the reviewer's raising this point to help us enrich the discussion!*
Dear Reviewer `ZrdC`, Thank you very much for your constructive feedback throughout the review, rebuttal, and discussion period. We're pleased that our discussions helped address your concerns, and we're thrilled to see the improved score. We will ensure that our discussions are fully incorporated into the final paper. Sincerely, The Authors of Paper 12439
Summary
This paper introduces WILDTEAMING, an automated red-teaming framework designed to enhance the safety of large language models (LLMs) by identifying and mitigating jailbreak tactics from user interactions. The framework consists of two main steps: mining in-the-wild user-chatbot interactions to discover novel jailbreak tactics and composing these tactics into more challenging adversarial attacks. The paper also presents WILDJAILBREAK, a large-scale open-source safety dataset generated through WILDTEAMING, containing 262K prompt-response pairs designed to improve the safety and robustness of LLMs.
Strengths
1. WILDTEAMING leverages real-world user interactions to identify and compose jailbreak tactics, providing a more diverse and realistic set of adversarial challenges than previous methods. 2. The creation of WILDJAILBREAK as an open-source resource is a significant contribution, offering a comprehensive dataset for safety training that addresses both harmful and benign queries. 3. The paper provides thorough evaluations of the framework's effectiveness, including comparisons with other state-of-the-art jailbreak methods and detailed analyses of the safety improvements achieved.
Weaknesses
1. While the evaluation is comprehensive, it primarily focuses on specific benchmarks and models. Broader evaluations across different types of LLMs and use cases might be necessary to generalize the findings. 2. The framework heavily relies on existing datasets and moderation tools, which may have inherent biases or limitations that could affect the results.
Questions
1. How well does WILDTEAMING generalize to LLMs beyond those tested in the paper? Have there been tests on a wider variety of models, including those with different architectures or training data? 2. How does the robustness of models trained with WILDJAILBREAK hold up over time as new adversarial tactics emerge? Are there mechanisms in place to continuously update the dataset and training methods? 3. What specific measures will be taken to prevent the misuse of the jailbreak tactics and the WILDJAILBREAK dataset?
Rating
5
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
1. The effectiveness of WILDTEAMING and WILDJAILBREAK might be influenced by the specific models and datasets used in the experiments, potentially limiting the generalizability of the results. 2. The framework's reliance on large-scale data mining and processing could be resource-intensive, potentially limiting its accessibility and applicability for smaller organizations or individual researchers.
Thanks much for the authors' thorough responses. Some of my concerns have been addressed. Kindly please find blow the remaining concerns: 1. Yes, I am interested. I would appreciate it if the authors could share the updated version. 2. I agree with the authors on the importance of diverse attacks in enhancing safety training. However, from the perspective of attacking, I still feel the point of "broadly revealing vulnerabilities" is weak. The proposed diversity lies in the different strategies of implementing the same harmful capability, instead of implementing different harmful capabilities. For example, to make the model teach how to make a bomb, malicious users seek for the most effective, instead of many, strategy to achieve the purpose. The only point of diversity in attacking is to bypass some over-optimized (for a subset of attacks) defense mechanism. However, given the observed marginal increase in ASR in tab. 2, it seems that the aforementioned point is not a major concern. 3. Thanks for providing the results of STL which convinces me the effectiveness of proposed safety alignment dataset. I also personally believe that learning an intrinsically safe model is promising and necessary for safe advanced AI. However, I don't think this means the other defense approaches like SmoothLLM [1] and Self-Reminders [2] (sorry for the missing citations in the initial review) should not be used and thus not attacked against. In fact, there are some arguments that safe LLM should be approached in a systematic way, e.g., the Llama Prompt Guard [3]. Therefore, I think the evaluation against these alternative defense methods is necessary to comprehensively demonstrate the effectiveness of proposed jailbreak attack method. [1] Robey et al., SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks, 2023 [2] Xie et al., Defending ChatGPT against jailbreak attack via self-reminders, Nature Machine Intelligence, 2023 [3] https://ai.meta.com/blog/meta-llama-3-1-ai-responsibility/
Thanks for the response. The response addresses most of my concern.
Decision
Accept (poster)