Mission Impossible: A Statistical Perspective on Jailbreaking LLMs

Summary

The paper introduces a new theoretical framework that views prompts as combinations of concepts and queries, allowing for a detailed analysis of how and why LLMs can be manipulated into producing unsafe responses. It proposes a statistical metric for alignment, focusing on quantifying the safety of the model's outputs. The study introduces an improved alignment strategy, Enhanced Reinforcement Learning from Human Feedback (E-RLHF), which modifies existing RLHF methods to increase the likelihood of safe responses from LLMs without additional training costs. Through experiments using standard benchmarks like AdvBench and HarmBench, the paper demonstrates that E-RLHF significantly outperforms traditional RLHF in resisting jailbreaking attempts, reducing the models' vulnerability to producing harmful outputs.

Strengths

The paper offers a unique statistical framework that conceptualizes input prompts into concepts and queries. This approach allows for a nuanced understanding of how LLMs process inputs and why they might generate unsafe outputs even under alignment efforts. The introduction of E-RLHF as an alignment strategy that doesn't require additional computational resources is a major strength. The empirical tests using established benchmarks such as AdvBench and HarmBench provide solid evidence that E-RLHF can significantly reduce the Attack Success Rate (ASR) compared to traditional RLHF.

Weaknesses

While E-RLHF improves alignment in controlled experimental conditions, its effectiveness in more complex conversational scenarios, where inputs can evolve over a series of interactions, remains untested.

Questions

The two datasets used in the work contain harmful requests about various concepts (e.g. drug, weapon). Have you tried to analyze the performance of E-RLHF on different concepts?

Rating

7

Confidence

3

Soundness

3

Presentation

2

Contribution

4

Limitations

Yes

Summary

The paper presents a statistical framework that provides a theoretical analysis of the jailbreaking problem in language models. The authors first examine the PAC-Bayesian bound to demonstrate that there is a non-trivial probability for LLMs to mimic harmful behavior if such information is present in the pre-training corpus. They then theoretically prove that a jailbreaking prompt can be found for such pretrained LMs and propose mitigating the jailbreak issue by modifying the harmful prompts.

Strengths

The paper's formulation and assumptions are clear and well-motivated. The proofs seem comprehensive and support the claims well. However, I don't have much background and PAC-Bayesian theory, so it's hard for me to verify if (1) the proofs are all rigorous (2) the proposed framework is truly novel instead of simple application of existing theorems. The experimental results from the proposed E-RLHF improve upon DPO-based fine-tuning.

Weaknesses

I find the connection between the theoretical part and section 5 (E-RLHF for actual experiment) quite handwavy. Expanding the safety zone is the claimed goal, but the proposed solution (through injecting safe concepts in a harmful prompt) seems really hacky. For example, as shown in Table 4, using safe prompt (4) gives much better results than safe prompt (1) for no reason. Overall, this proposed method seems like a small trick that the authors try to incorporate into the paper just to show some empirical value of their theoretical analysis. I also find that with such modifications, the MT-Bench score (briefly mentioned in line 355) is lower under E-DPO than DPO. Lastly, this might be a biased opinion (that's why I didn't factor this part into my scoring decision and still gave a marginally above accept score): I feel that after all the theoretical proofs (despite their elegance), the conclusion (LLMs will mimic toxic behavior if such toxic content is present in the training corpus, jailbreak is unpreventable under reasonable assumptions, and by expanding the safe zone we reduce the probability of jailbreaking) is very intuitive and does not provide much additional insight into the problem. Maybe the mathematical formalization of the jailbreak problem itself is meaningful, and I will let the other reviewers judge the novelty of such a framework.

Questions

Could you provide more details on the MT-bench's performance? For example, you have an ablation study on different safety prefixes' effects in Table 4. What about their impact on MT-bench? In line 327 and the ablation study in the appendix, you show that non-harmful prompts should not be modified (otherwise, the model performs much worse). Could you relate this phenomenon to the safe/harmful zone or your theoretical analysis? I don't see why adding a safe prefix to a non-harmful prompt negatively impacts the safe/harmful zone, and under your assumption, this should not lead to worse behavior.

Rating

6

Confidence

2

Soundness

2

Presentation

3

Contribution

3

Limitations

No concerns on the limitations

Summary

The paper addresses a very important issue of our time, the safety of LLMs. LLMs are already used in various applications and will be present in more applications to come, as e.g. Microsoft, Apple and Google are integrating LLMs in their applications and operating systems. Hence, the question on how to make the systems more robust against jailbreaking is a very important question. This paper offers a statistical approach to this question and presents experiments. The mathematics is based PAC-Bayesian approach and builds upon a very nice formalization of splitting a prompt into queries and concepts.

Strengths

The mathematics are presented very well. In particular, the intuitions and interpretations of the mathematical concepts are presented well, and can also be followed by readers, who are not able to follow all the details of every equation. Overall, I really appreciated the formal approach to modelling jailbreaking and the conclusions, that it is impossible to avoid jailbreaking.

Weaknesses

There is a disconnect between the theory and the experiments. The definitions, theorems, etc. are all presented and motivated very nicely, but could also apply to any other mapping of a stochastic system. Oversimplified, one may state that there is a mapping from an input space into an output space, in which the output space can be divided into desired and undesired outputs. The goal is to reduce the space of undesired outputs. To make this a case specifically about LLMs, there needs to some form of conclusions that feed into the experiments. Unfortunately, I don't see this connection. The experiments use a systems prompt that is preprended to the actual prompt. I do not see, how this follows from the theory. This is really unfortunate, because I would really like to see, how the experiments connect to the theory.

Questions

Could you please point out the connection between experiment and theory that I have seemed to miss.

Rating

4

Confidence

2

Soundness

3

Presentation

3

Contribution

2

Limitations

I don't see any negative social impacts. As stated above, I would like to see a stronger connection between experiments and theory.

Summary

The paper provides a theoretical insight about LLM jailbreaks using PAC-Bayesian bound for pretraining LLMs. It assumes that there always exists the harmful data in the mixture, and as the model is trained on this mixture, the model will probably produce the responses in harmful zone (it has a specific definition in the main paper). Based on this framework, the authors suggests that the safety zone of the models should be extended, and to this end, they introduce a method called E-RLHF which can expand the safety zone. E-RLHF replaces the harmful prompts x_h into benign prompts x_s, and replace some of the terms in RLHF (and DPO) to make sure that the model keeps in the safety zone. Experimental results also show that it does not sacrifice the general capabilities but can be improved in the safety perspective.

Strengths

- This paper suggests a theoretical insight about the LLM jailbreaks, which were not addressed much in the previous literature on jailbreaking. The theoretical framework is sound and compelling. - Based on this framework, the paper also suggests a simple training trick that can lead to better safety training. - Empirical results also show that their idea is working well, shown by Harmbench, AdvBench, and MT-Bench scores. - The paper provides extensive evaluation results on jailbreaking setups, providing the results from more than 10 attack setups.

Weaknesses

- E-RLHF is an inaugural and simple form of expanding the safety zone of LLM; I think there could be more sophisticated and effective ways, and I hope the authors will address this in the future works. - Other than that, I think there is no big weakness in the paper, but have some minor comments: - Eq (2): it is slightly confusing that in D_KL, the term only have p_LM(x) and p_SFT(x), not p_LM(e|x) and p_SFT(e|x). - About writing: I am not familiar with using the term "explanation" -- instead, I think using "response" is more common. At the first glance, it was hard to comprehend the meaning of the term.

Questions

No specific questions about the paper.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors provided limitations section in the paper.

Official Comment by Reviewer UXGw

Dear Authors, Thanks for providing additional clarifications to the points I asked in the review. After reading the response (for both my review and others), I decide to increase the score (from 5) to 6.

Answer

Dear authors, Thank you very much for the lengthly reply to my main concern. The reply validates, that I did understand the main points of the paper and approach. Unfortunately, I don't see any additional arguments, that directly answer my question. I do see the importance of the experiments and results and I also do so see the theory, that you discuss in the paper. I just don't see, how the two are connected. Prompt engineering and system prompts are a well established method in many LLM-applications. They do not follow from your theory as there are other ways to change the safe and unsafe areas. I appreciate the answer, but will remain with my initial assessment of the paper.

Response to comments

Dear reviewer qvRD, We appreciate the acknowledgements of our work and thanks for responding to our rebuttal!