Jailbroken: How Does LLM Safety Training Fail?

Summary

This paper offers an insightful examination of adversarial misuse, or "jailbreak" attacks, against large language models (LLMs) such as OpenAI's GPT-4 and Anthropic’s Claude v1.3. By analyzing two proposed failure modes of safety training—competing objectives and mismatched generalization—the authors provide an empirical study of why these attacks can succeed and how they might be created. Nevertheless, this paper could be critiqued for a lack of novel methodological or technical insights.

Strengths

This paper provides a comprehensive examination of the vulnerabilities in large language models (LLMs) to "jailbreak" attacks.

Weaknesses

I enjoyed reading the extensive experimental results and inspiring observations. The main reason I did not raise a higher score is that the paper seems to lack rigor in defining key concepts and quantifying its statements. For example, the paper did not precisely define what a jailbreak attack is in the context of LLM. It is unclear what is safe/unsafe since that can be quite subjective. In making "analysis", e.g., "Competing Objectives" in Section 3.1, the paper only provided selected examples rather than formalism. I feel the empirical studies are not sufficient to publish because there have been many existing empirical studies on jailbreaks and this paper does not distinguish itself from the rest with more in-depth formalism. However, I could be biased. So I can adjust scores after reading other review comments and authors' responses.

Questions

none

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

technical insights or novel method

Summary

The paper studies jailbreak attacks. More specifically authors hypothesize two failure modes of LLMs 1. competing objective and 2. mismatched generalization and base their discussion on why jailbreak attacks succeed based on these hypothesis made. They then quantitatively perform experiments on different jailbreak attacks and report the success of each.

Strengths

1. The paper studies a timely and important topic. 2. The observations/discussion on competing objectives was interesting.

Weaknesses

1. I would have liked to see more in depth and concrete discussions on what could be done to improve these systems against jailbreak attacks and even maybe the execution of this idea in small scale would have made the paper very strong. 2. The dataset discussion in section 2.2 could be more organized and clear with more detailed discussions to increase clarity. 3. I felt like the results do not directly support claims made in the paper in the sense that jailbreaks happen due to the two reasons 1.competing objectives And 2.mismatched generalization hypothesized by the authors. The quantitative results mostly show the success of the jailbreaks which is a nice finding on its own and mostly known, but they do not directly support the claims that these attacks are successful and happen due to competing objectives and mismatched generalization. Moreover, there might be other contributors not considered in this work. Although authors provide discussions and tie these jailbreaks to the hypothesized reasons in section 3, in my opinion analysis in section 4 does not strongly and directly support those claims. Maybe the paper needs to be motivated in some other way or written in a different manner so that claims are supported and reflected accurately in the experimental discussions and results. **Minor comments:** Typo line 288 ( such an attack is technically is beyond the scope of our threat model -> such an attack is technically beyond the scope of our threat model).

Questions

Addressing my first and third concerns in the weaknesses section would be good.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

Authors provided a discussion on limitations and broader impacts of their work which is appreciated.

Summary

This manuscript provides an initial exploration of the robustness of LLM systems against adversarial misuses, specifically focusing on "jailbreak" attacks. The authors successfully summarize existing threats, propose plausible hypotheses, and conduct empirical evaluations on three LLMs: GPT-4, Claude v1.3, and GPT-3.5. They identify two failure modes in the current safety training: Competing Objectives and Mismatched Generalization. The paper concludes by deriving valuable defense implications from their analysis and evaluations. Overall, this submission initiates a constructive discussion on the development of safe LLM systems.

Strengths

The paper addresses a timely and significant topic. The authors carefully examined the problem, presented their hypotheses, and supported them with well-designed empirical experiments. Helpful defense implications are also provided following the analyses.

Weaknesses

Certain sections of the writing could be refined to improve comprehension for a general audience. For example, the introduction to the current safety training mechanism and the general LLM training framework lacks sufficient detail, which may hinder readers' understanding of the concepts of Competing Objectives and Mismatched Generalization. Furthermore, the significance of the paper is constrained by the absence of direct access to detailed information regarding LLMs they use in their paper.

Questions

1. Section 2.2 primarily focuses on the detailed description of the models and datasets used in the authors' evaluation. Will it be better placed in Section 4. 2. There appears to be some overlaps between Competing Objectives and Mismatched Generalization. For example, both the use of encoded output in Base64 and requesting unusual output formats may exploit prefix injection. Can you better explain the relationship between these two failure modes? 3. The description of the Adaptive Attack in Section 4.1 lacks clarity.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The author discussed the limitation arising from the lack of detailed information on LLMs, thus their hypotheses cannot be directly confirmed.

Summary

The paper evaluates GPT models and Claude models against Jailbreaks and comes up with two possible explanations for why jailbreaks are successful- 1. competing objectives between pretraining + IF finetuning and safety finetuning; and 2. pretraining+finetuning generalizing better than safety tuning. It also argues for safety-capability parity and that we need to move beyond the pretraining + post-training for safety paradigm to come up with more sophisticated methods for safety training.

Strengths

Strengths: 1. Very clearly and well written; enjoyable read 2. Makes intuitive+compelling hypotheses around why jailbreaks are successful and makes them explicit and clear; provides evidence supporting the hypotheses 3. The safety-capability parity claim the paper makes is important and it's presented well with evidence for why its needed 4. I think the paper is significant given that it makes a compelling argument for why the current paradigm of safety finetuning won't scale and with good evidence for it. The evaluations and hypotheses are quite intuitive + based on existing work but I think the presentation of this work in the paper and the narrative it creates is compelling and important.

Weaknesses

Weaknesses: 1. Evaluations could be more robust and have a higher quantity of prompts 2. The main claims and methods are interesting and well-presented but not particularly surprising / novel 3. It should make clear that the two possible hypotheses are two reasons for jailbreaks but not the entire surface area of possible reasons

Questions

It would be great if more data is added to evals and the evals are made transparent + available for everyone

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Yes

Summary

This paper analyzes why safety training of LLMs fail and what could be the root causes for them. They offer two explanations: competing objectives and inadequate coverage of the model capabilities during safety training. They categorize existing attacks and propose some new ones into these two buckets, and evaluate GPT-4 and Claude v1.3. The results show that both the models are susceptible to jailbreaks. Post-rebuttal comment: The authors addressed my concerns. I am happy to keep my score of weak accept.

Strengths

I am broadly in agreement with the explanations put forth in this paper. The paper is written quite precisely and in a convincing manner. The experiments show that most of the 317 prompts constructed can be used in combination with one of the methods to successfully jailbreak.

Weaknesses

The paper is mainly about evaluation of the models and the difficulty ahead of the defenses. While it is informative, it would have strengthened the paper to try some experiments and ablations on safety training (e.g., trying to remedy the mismatch in pretraining and safety-tuning in some ways). I can imagine the difficulty of experimenting with large, closed-source models, but some experimentation with small/medium open-source models would have helped. My other concern is how much of the material in the paper is novel. Several of the existing attacks may have identified these or equivalent root causes and I am unable to clearly estimate how much of the paper's content is novel/surprising.

Questions

What specific explanations and observations in the paper are novel when the existing attacks are taken in account? Do you have any analysis of safety training interventions based on your identification of the key limitations of the current methods?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors have discussed the limitations and societal impact. They have also reported that they have disclosed their findings to the model authors and will coordinate the future release of the specific attacks with them to avoid harm. I appreciate these efforts.

Summary

The paper studies jailbreaking of large language models. The authors identify two categories of causes of jailbreaks, competing objectives and mismatched generalization, and use this insight to analyze existing jailbreaks and construct new ones. They empirically study the effectiveness of these jailbreaks on state-of-the-art models that have been trained to refuse unsafe instructions, and draw several conclusions from their analysis.

Strengths

Jailbreaks pose an important safety problem for publicly-deployed AI systems. The conceptual analysis is novel and insightful, and the experiments are careful and thorough. The conclusion about safety-capability parity is especially interesting. Moreover, merely demonstrating extremely successful attacks directly helps companies deploying models to make their models safer via responsible disclosure. The paper is also very well-written.

Weaknesses

The main weakness of the paper, which is easily rectified, is that the authors do not clarify (unless I somehow missed it) which if any models were queried as part of the construction of the attacks, or more broadly if the attacks have any particular intended target model. It seems reasonable to query a model when constructing an attack since the threat model allows this, but the information should be provided as it is needed to properly interpret the results of the paper. I don't think anything long and detailed is necessary if the process was somewhat ad-hoc, but some indication is necessary I think. As a case in point, the paper interprets the results on GPT-3.5 (Table 3) as "GPT-3.5 not having the capability to understand complex inputs" (p. 8 line 336). However, I suspect that a more likely explanation of why the top 4 attacks are all attacks from jailbreakchat.com is that they specifically targeted at GPT-3.5 (since that version of ChatGPT is free-to-use and hence likely much more popular). I also disagreed slightly with some of the claims in the section "What Scaling Won't Solve" (p. 8 line 350). The authors argue that scaling will not solve the problem of competing objectives, since the objectives will still compete. However, larger models may be able to obtain Pareto improvements on both objectives. Hence even though the competition between the objectives may remain, the models may still perform as well as necessary on the safety objective. Nevertheless, I did agree with the argument that scaling alone will not solve the problem of mismatched generalization (not counting using the models to generate stronger attacks, which is discussed later). Minor point: in the phrase "targeted training is insufficient" (p. 8 line 325), the term "targeted training" is not defined. I initially interpreted it to mean "adversarial training" (which the results do not demonstrate is necessarily insufficient), whereas I think the authors mean "adversarial training that only targets a subset of the possible failure modes" (which it is unsurprising is insufficient, though I suppose it might still be worth noting).

Questions

No questions other than those given above

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

No limitations other than those given above

Summary

The paper investigates two reasons why jailbreak attacks against SOTA LLMs (GPT-4, GPT-3.5 Turbo, Claude v1.3) succeed despite extensive safety training: a) competing training objectives (safety objectives vs. pretraining/instruction tuning) and failure of safety training to generalize to conditions covered by pretraining/instruction training but not safety training. These two modes are then used as guiding principles to design new jailbreak attacks that are empirically evaluated against the above three models, starting from both known and newly synthesized harmful prompts; a large number of these attacks are shown to have a high success rate. The authors provide concluding hypotheses regarding future work to defend against attacks (more sophisticated safety models that match the models' basic capabilities).

Strengths

The paper aims to examine jailbreak attacks in a more principled way and presents an empirical evaluation that is broader in scope than previous studies. Though the insights are not entirely surprising, the paper does a good job explaining and exemplifying the two failure modes, evaluating SOTA models along those axes, and providing quantitative results. Although the failure modes are intuitively clear and have been known informally by most developers of LLMs, it is valuable to see a concise formulation of the problems and a fairly thorough experimental study.The paper offers high-level suggestions for how to do red-teaming or safety training more effectively and as such would be valuable for the community. The presentation is very clear; authors observe responsible disclosure practices and have discussed potential ethical considerations (recipes for creating jailbreak attacks).

Weaknesses

1. The paper focuses on two failure modes only. Arguably these could be the most important ones, but the bigger question is whether one could come up with a more comprehensive taxonomy of failure modes. The paper doesn't discuss this in more detail. 2. Many of the observations and conclusions are by necessity tentative and unconfirmed, due to black-box only access to models. The authors could have verified their hypotheses more directly by using a smaller open-source model where more details about training conditions and data resources are available. This setup could also have been used to explore the space of failure modes more systematically (e.g., what role does decoding play?). 3. The most successful attacks were combinations, but those were not studied in detail from a conceptual point of view- this seems to merit an entirely separate discussion: since there's a combinatorial space of combined attacks, how could models be trained successfully to defend against these? 4. The 'safety-capability' parity point remains vague - what does this mean for the actual safety model, are there concrete mechanisms you could suggest to make models more 'sophisticated'? It would be good to see more discussion of at least points #1 and #3 in the paper.

Questions

It's not clear to me where the attacks from jailbreakchat.com (p. 7) land in Table 1 -- which ones does this map to?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors discuss the limitations (e.g., no access to models themselves, hence tentative conclusions) but in the light of these they could have opted to also include some targeted experiments with open-source models (relegated to future work in the paper).