Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks

Despite advances in AI alignment, large language models (LLMs) remain vulnerable to adversarial attacks or jailbreaking, in which adversaries can modify prompts to induce unwanted behavior. While some defenses have been proposed, they have not been adapted to newly proposed attacks and more challenging threat models. To address this, we propose an optimization-based objective for defending LLMs against jailbreaking attacks and an algorithm, Robust Prompt Optimization (RPO) to create robust system-level defenses. Our approach directly incorporates the adversary into the defensive objective and optimizes a lightweight and transferable suffix, enabling RPO to adapt to worst-case adaptive attacks. Our theoretical and experimental results show improved robustness to both jailbreaks seen during optimization and unknown jailbreaks, reducing the attack success rate (ASR) on GPT-4 to 6% and Llama-2 to 0% on JailbreakBench, setting the state-of-the-art. Code can be found at https://github.com/lapisrocks/rpo

Paper

References (82)

Scroll for more · 38 remaining

Similar papers

Peer review

Reviewer EVpE8/10 · confidence 5/52024-07-05

Summary

This paper introduces a new approach called Robust Prompt Optimization (RPO) for defending large language models (LLMs) against jailbreaking attacks. The key contributions are: 1. Formalizing a minimax optimization objective for ensuring safe LLM outputs under a realistic threat model involving various attacks and adaptive adversaries. 2. Proposing the RPO algorithm, which directly optimizes for the defense objective using principled attack selection and discrete optimization. 3. Developing an easily deployable suffix-based defense that achieves state-of-the-art performance in protecting LLMs against jailbreaks on benchmark datasets. The RPO method works by optimizing a set of "trigger tokens" that enforce safe outputs even under adversarial attacks. The authors evaluate RPO on recent red-teaming benchmarks and show it significantly reduces attack success rates on models like GPT-4 and LLaMA-2. Key advantages of RPO include its negligible inference cost, minimal impact on benign prompts, and ability to transfer to black-box models and unknown attacks. The paper provides both theoretical analysis and experimental results demonstrating RPO's effectiveness as a robust, universal defense against various jailbreaking techniques.

Strengths

Originality: - Proposes the first formal optimization objective for defending language models against jailbreaking attacks, incorporating the adversary directly into the defensive objective. This is a novel formulation of the problem. - Introduces Robust Prompt Optimization (RPO), a new algorithm to optimize for this defensive objective using a combination of attack selection and discrete optimization. Quality: - Provides theoretical analysis showing that optimizing their proposed objective is guaranteed to improve robustness, even on unseen instructions and attacks. This gives a solid theoretical grounding. - Conducts extensive empirical evaluation on recent benchmarks (JailbreakBench and HarmBench), demonstrating state-of-the-art performance in reducing attack success rates. - Shows transferability of the defense to black-box models like GPT-4 and resistance to adaptive attacks, indicating the approach is robust. Clarity: - The paper is generally well-structured and clearly written. - Key ideas and contributions are summarized concisely in the introduction. - The methodology is explained step-by-step with supporting equations and an algorithm description. Significance: - Addresses an important problem in AI safety - defending large language models against jailbreaking attacks that could lead to harmful outputs. - Achieves state-of-the-art results on reducing attack success rates (down to 6% on GPT-4 and 0% on Llama-2). - The proposed defense is lightweight and easily deployable as a suffix, making it practical for real-world implementation. - The approach is model-agnostic and transfers well to different LLMs, including closed-source models, increasing its potential impact. Overall, this paper makes significant contributions in formulating and addressing the challenge of defending LLMs against jailbreaking attacks. The combination of theoretical grounding, novel algorithmic approach, and strong empirical results on challenging benchmarks makes this work quite impactful for the field of AI safety and robustness.

Weaknesses

1. Limited discussion of computational costs: - The paper doesn't provide details on the computational resources required for RPO optimization. - It's unclear how long it takes to generate the defensive suffix or how this scales with different model sizes. - Actionable improvement: Include a section on computational requirements, comparing RPO's runtime and resource usage to existing defenses and baseline LLM inference. 2. Lack of ablation studies: - The paper doesn't explore the impact of different components of RPO (e.g., attack selection frequency, batch size, number of iterations). - Actionable improvement: Conduct ablation studies to show how each component contributes to the overall performance and to guide practitioners in tuning these hyperparameters. 3. Limited exploration of potential negative impacts: - While the paper focuses on defending against harmful outputs, it doesn't discuss potential unintended consequences of the defense mechanism. - For instance, could RPO inadvertently block legitimate but sensitive queries? - Actionable improvement: Include a section on potential limitations and negative impacts, with empirical analysis on false positive rates for benign but sensitive queries. 4. Insufficient comparison to other optimization-based defenses: - While the paper compares to some existing defenses, it doesn't thoroughly compare to other optimization-based approaches in adversarial robustness literature. - Actionable improvement: Include comparisons to adversarial training methods adapted for language models, such as those proposed by Ziegler et al. (2022) in "Adversarial Training for High-Stakes Reliability". 5. Limited exploration of transfer learning: - While the paper shows transfer to GPT-4, it doesn't explore how well the defense transfers between models of different sizes or architectures. - Actionable improvement: Conduct experiments on transfer learning between models of varying sizes (e.g., from smaller to larger models) and different architectures (e.g., from decoder-only to encoder-decoder models). 6. Lack of human evaluation: - The paper relies primarily on automated metrics for evaluation. - It's unclear how the defended model's outputs are perceived by human users in terms of safety and quality. - Actionable improvement: Conduct a human evaluation study to assess the perceived safety and quality of outputs from models with and without RPO defense. 7. Limited discussion on the choice of loss function: - The paper uses log probability as the loss function but doesn't justify this choice or explore alternatives. - Actionable improvement: Provide a justification for the chosen loss function and experiment with alternative loss functions (e.g., KL-divergence, earth mover's distance) to see if they yield better results. 8. Insufficient analysis of the learned defensive suffixes: - The paper doesn't provide an in-depth analysis of the structure or content of the learned defensive suffixes. - Actionable improvement: Include a section analyzing the learned suffixes, perhaps using interpretability techniques to understand what patterns the defense is learning. 9. Limited exploration of multi-turn interactions: - The paper focuses on single-turn interactions, but many real-world scenarios involve multi-turn dialogues. - Actionable improvement: Extend the evaluation to multi-turn scenarios to assess how well the defense holds up over extended interactions. 10. Lack of discussion on potential adaptive attacks: - While the paper mentions resistance to adaptive attacks, it doesn't explore specific adaptive strategies an attacker might employ against RPO. - Actionable improvement: Include a section on potential adaptive attacks against RPO and empirically evaluate the defense's performance against these hypothetical attacks. These specific improvements would strengthen the paper's contribution and provide more comprehensive insights into the proposed defense mechanism.

Questions

1. Computational Resources and Scalability: Question: What are the computational requirements for RPO optimization? How does the runtime scale with model size and dataset size? Suggestion: Provide a detailed analysis of computational costs, including time and hardware requirements for different model sizes. 2. Hyperparameter Sensitivity: Question: How sensitive is RPO to its hyperparameters (e.g., attack selection frequency, batch size, number of iterations)? Suggestion: Conduct and present an ablation study showing the impact of different hyperparameter choices on the defense's effectiveness. 3. False Positive Analysis: Question: Does RPO inadvertently block legitimate but sensitive queries? What is the false positive rate? Suggestion: Perform an analysis on a set of benign but potentially sensitive queries to assess any unintended blocking. 4. Comparison to Adversarial Training: Question: How does RPO compare to adversarial training methods adapted for language models? Suggestion: Include a direct comparison with adversarial training approaches, particularly those designed for language models. 5. Transfer Learning Capabilities: Question: How well does the RPO defense transfer between models of different sizes or architectures? Suggestion: Conduct experiments showing transfer performance between various model sizes and architectures. 6. Human Evaluation: Question: How do human evaluators perceive the safety and quality of outputs from RPO-defended models compared to undefended models? Suggestion: Conduct a human evaluation study and present the results. 7. Loss Function Choice: Question: Why was log probability chosen as the loss function? Have alternative loss functions been considered? Suggestion: Provide justification for the chosen loss function and experiment with alternatives like KL-divergence or earth mover's distance. 8. Analysis of Learned Suffixes: Question: What patterns or structures are present in the learned defensive suffixes? Suggestion: Perform an in-depth analysis of the learned suffixes, possibly using interpretability techniques. 9. Multi-turn Interactions: Question: How does RPO perform in multi-turn dialogue scenarios? Suggestion: Extend the evaluation to include multi-turn interactions to assess the defense's effectiveness over extended conversations. 10. Adaptive Attacks: Question: What specific adaptive strategies might an attacker employ against RPO, and how does the defense perform against these? Suggestion: Outline potential adaptive attacks and empirically evaluate RPO's performance against them. 11. Impact on Model Performance: Question: Does the RPO defense impact the model's performance on non-adversarial tasks? Suggestion: Evaluate the defended model on standard language modeling benchmarks to assess any potential degradation in general performance. 12. Generalization to Other Types of Attacks: Question: How well does RPO generalize to types of attacks not seen during training? Suggestion: Test the defense against a held-out set of novel attack types not used during optimization. 13. Ethical Considerations: Question: Are there any potential ethical issues or misuse scenarios associated with deploying RPO in real-world applications? Suggestion: Include a discussion on the ethical implications and potential misuse of the technology. 14. Integration with Existing Systems: Question: How easily can RPO be integrated into existing language model deployment pipelines? Suggestion: Provide guidelines or a case study on integrating RPO into a typical LLM deployment setup. 15. Longevity of the Defense: Question: How long is the RPO defense expected to remain effective as new attack methods are developed? Suggestion: Discuss the expected longevity of the defense and propose strategies for keeping it up-to-date with evolving attacks. These questions and suggestions aim to address key aspects of the paper that could benefit from further clarification or exploration, potentially changing opinions on the work's impact and completeness.

Rating

8

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, the authors adequately addressed the limitations.

Authorsrebuttal2024-08-12

Follow up to reviewer

Dear Reviewer EVpE, We are thankful for your review. With the rebuttal deadline drawing near, please inform us if your concerns have been properly addressed. We are ready to offer further clarification. Best regards, The Authors

Reviewer YYJE7/10 · confidence 2/52024-07-12

Summary

This paper introduces Robust Prompt Optimization (RPO), a novel method for defending LLM against jailbreaking attacks, which manipulate prompts to induce harmful behavior. Inspired by the Adversarial Training, RPO optimizes a suffix for the LLM prompt, ensuring safe responses even when the input is modified by an attacker. The RPO method contains two steps, jailbreaking prompt selection and discrete optimzation, which is also based on a complete theoretical proof. RPO demonstrates significant improvements in attack success rate (ASR) on various LLMs, including GPT-4, and is transferable to unknown attacks, making it a robust and versatile defense mechanism for LLMs.

Strengths

**Good Innovation**: Optimization-based methods are widely used for optimizing jailbreak suffixes, but this paper innovatively applies it to the reinforcement of security alignment, introducing a new strategy for jailbreak defense. **Solid Theoretical Foundation**: The paper conducts a detailed theoretical analysis of RPO (Reinforced Protection Optimization). **Elaborate Empirical Evaluation**: The effectiveness of the method is demonstrated through extensive empirical experiments.

Weaknesses

**Transferability Discussion**: While RPO shows promising transferability, which is very interesting, the paper does not seem to discuss this aspect sufficiently. I recommend that the authors further explore and elaborate on why RPO exhibits good transferability. **Efficiency Concerns**: Section 4.3 discusses the method only adds 20 additional tokens, which does not significantly increase the cost of inference for benign prompts. However, there appears to be a lack of discussion on the efficiency of the optimization process itself. It is well-known that token-level optimization, typified by methods like GCG, is often criticized for its high optimization costs. Thus, I am concerned about the optimization efficiency of RPO.

Questions

1.How many rounds of optimization are required for a successful optimization of RPO, and how much time does each round typically take? 2.Why do the optimized defense suffixes exhibit good transferability across different models?

Rating

7

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper thoroughly discusses the existing limitations.

Authorsrebuttal2024-08-12

Follow up to reviewer

Dear Reviewer YYJE, We are thankful for your review. With the rebuttal deadline drawing near, please inform us if your concerns have been properly addressed. We are ready to offer further clarification. Best regards, The Authors

Reviewer yUwN6/10 · confidence 5/52024-07-12

Summary

The paper proposes a novel method, Robust Prompt Optimization (RPO), to enhance the robustness of large language models (LLMs) against jailbreaking attacks. Existing defenses, which operate during pre-training or fine-tuning stages, incur high computational costs. The proposed RPO method optimizes a lightweight suffix at the inference stage, incorporating the adversary into the defensive objective. Theoretical and experimental results demonstrate that RPO reduces the attack success rate on GPT-4 and Llama-2 models, setting a new state-of-the-art in defense against jailbreaking attacks.

Strengths

1. The proposed method demonstrates substantial improvements in reducing attack success rates, outperforming existing defense mechanisms. 2. Unlike other methods that operate during pre-training or fine-tuning stages, RPO is computationally efficient, operating during the inference stage without significant overhead. 3. The authors provide a thorough theoretical analysis of the defense method.

Weaknesses

1. The method can not address other failure modes such as deception and malicious code generation, limiting its applicability. 2. How does the proposed RPO method handle the evolution of attack strategies over time, and how frequently would the defensive suffix need updating? The authors should have provided more analysis. 3. What are the potential trade-offs between robustness and model performance when applying RPO to different LLM architectures? The authors should have provided more analysis.

Questions

Please refer to Weaknesses for details.

Rating

6

Confidence

5

Soundness

4

Presentation

4

Contribution

3

Limitations

Yes.

Authorsrebuttal2024-08-12

Follow up to reviewer

Dear Reviewer yUwN, We are thankful for your review. With the rebuttal deadline drawing near, please inform us if your concerns have been properly addressed. We are ready to offer further clarification. Best regards, The Authors

Reviewer Md6V7/10 · confidence 4/52024-07-13

Summary

In this paper, the authors proposes a new defense, called Robust Prompt Optimization(RPO) to defend the jailbeak attack. It optimizes a secure suffix with min-max optimization. Experiments reveal that it can defend multiple existing attacks

Strengths

1 This paper is well written. 2 The authors give theoretical proof to explain the working dynamics of RPO. 3 The proposed method is simple and easy to understand.

Weaknesses

1 RPO obtains high ASR against the JBC attack on Vicuna which is much worse than the Rephrasing defense. 2 Although RPO can save a lot of computation cost than other methods during the inference time. The cost to generated the suffix can not be simply ignored. From my experimence, it usually needs multiple hours even on a single A100 GPUs. Regarding this point, self-reminder [1] and ICD [2] might be a better choice. 3 In Table 1, authors only report results on two open-source and closed-source models. More experiments are needed to fully investigate the capability of RPO, such as Vicuna-7B, Llama-13B, QWen-7B and claude. 4 Although in the related work section, the authors list a lot of current defenses. However, in Table 1, they compare RPO with only a few. I would suggest authors compared DPO with stronger defenses such as [3] , [4] and [5].

Questions

See weakness.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Authors well addresses the limitations of the proposed method. [1] Defending chatgpt against jailbreak attack via self-reminder [2] Jailbreak and guard aligned language models with only few in-context demonstrations [3] Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations [4] Defending large language models against jailbreaking attacks through goal prioritization. [5] Rain: Your language models can align themselves without finetuning

Authorsrebuttal2024-08-12

Follow-up to reviewer

Dear Reviewer Md6V, We are thankful for your review. With the rebuttal deadline drawing near, please inform us if your concerns have been properly addressed. We are ready to offer further clarification. Best regards, The Authors

Reviewer Md6V2024-08-13

Dear authors, I am happy that you address most of my concern. 1 However, considering defending jailbreak attacks from prompt tuning view is not a full new direction. For example, related works [1] and [2] study the same problem and propose the corresponding methods. I list them here is not doubting the novelty of this paper. However, discussing with them might help readers understand the importance of the prompt-based defenses better. 2 Notice that you release the code. How many iterations do RPO need for its convergence? [1] Fight Back Against Jailbreaking via Prompt Adversarial Tuning [2] On prompt-driven safeguarding for large language models

Authorsrebuttal2024-08-13

Response to reviewer

We thank the reviewer for the response and are glad most concerns are addressed. > *Discussing related work* We are glad the reviewer has brought these works to our attention and agree they should be discussed. 1. PAT [1] is concurrent to our work, and explores a similar optimization objective. However, PAT is only optimized on the GCG attack, which may limit its transferability to newer attacks. 2. DRO [2] also explores prompt-based defenses for LMs but optimizes a soft prompt in embedding space. This prevents DRO from transferring to closed-source models. In addition, the DRO objective is only a minimization objective, not a min-min optimization objective like RPO, which incorporates the adversary into optimization. This makes RPO more adaptive to jailbreaking attacks. We will include baseline comparisons and a more detailed discussion in the revised version. >*How many iterations do RPO need for its convergence?* In practice, we find that RPO requires around 100 iterations to converge on a single A100 GPU (around an hour), and we optimize our suffixes for 500 iterations, which takes 4-5 hours. In the revised version, we will conduct a more detailed analysis of performance over time. We are happy to further clarify, and hope the reviewer will raise the score if all concerns have been addressed.

Authorsrebuttal2024-08-14

Response to reviewer

Thank you for pointing this out. PAT and DRO indeed also optimize on benign prompts. However, from the reported MT-Bench scores in [1], RPO is competitive with PAT on FPR. We provide the results here, where the MT-Bench scores are taken directly from [1]. | Model | Method | MT-Bench | |-------------|--------|----------| | Vicuna-13B | Base | 6.57 | | | PAT | 6.15 | | | RPO | 5.96 | | GPT-3.5 | Base | 8.32 | | | PAT | 8.06 | | | RPO | 7.81 | | GPT-4 | Base | 9.32 | | | PAT | 8.77 | | | RPO | 9.20 | On weaker models such as Vicuna and GPT-3.5, RPO has a more significant effect on benign prompts than PAT, but on more capable models such as GPT-4, it has a smaller effect than PAT. This may be due to optimizing RPO on semantically meaningful harmful prompts. We will conduct a more detailed analysis and comparison in the revised version. Including benign prompts in the optimization of RPO is an interesting suggestion that may further reduce benign impact. We will try this improvement as an ablation in the revised version. For future work, modifying RPO to optimize semantically meaningful prompts, such as constraining candidate tokens to low perplexity ones, may also improve this. [1] Mo, Y., Wang, Y., Wei, Z., & Wang, Y. (2024). Fight Back Against Jailbreaking via Prompt Adversarial Tuning.

Reviewer Md6V2024-08-14

Thank you for your reply. I will increase my score to 7. Can not wait to see the change in the revised version of this paper.

Authorsrebuttal2024-08-14

Thanks for appreciating our rebuttal and increasing the score! We will add all the new results and discussion in the revision.

Sungyoon Lee12025-09-28

The algorithm says $$A^\ast =\arg\min_{\mathcal{A}} \mathcal{L}\_j^{\text{adv}}\sum\_{1\leq o\leq m} (A_o(x^{(j)})).$$ So the loss is evaluated with the ``summed inputs'' $\sum\_{1\leq o\leq m} (A_o(x^{(j)}))$. How do we take the summation over discrete inputs? It seems weird. I think it should be $$A^\ast =\arg\min_{A\in\mathcal{A}} \sum\_{1\leq j\leq m}\mathcal{L}\_j^{\text{adv}} (A(x^{(j)}))$$ or something else.

Program Chairsdecision2024-09-25

Decision

Accept (spotlight)

© 2026 NYSGPT2525 LLC