Fight Back Against Jailbreaking via Prompt Adversarial Tuning

Summary

In this paper, the authors introduce a new defense mechanism called Prompt Adversarial Tuning (PAT) designed to protect LLMs from jailbreak attacks. PAT enhances the robustness of these models by attaching a defensive prompt control to user inputs, optimized through a combination of adversarial and benign prompts. This method reduces the success rate of advanced attacks to nearly zero while incurring minimal computational overhead. However, there are some concerns. The main optimization idea is relatively straightforward and bears similarities to the GCG [1] approach. Additionally, the robustness of the experimental results is questionable, particularly due to the use of prefix-based methods to calculate the ASR. [1] Universal and Transferable Adversarial Attacks on Aligned Language Models.

Strengths

1. PAT introduces a new system-level defense mechanism that is interesting and does not alter the LLM itself. 2. PAT demonstrates good transferability across different models, both open-source and closed-source, showcasing its versatility. 3. The method preserves the model's performance on benign tasks while providing robust defense against malicious prompts.

Weaknesses

1. The optimization of the defense prompt incurs computational overhead, and the optimization process using greedy sampling is time-consuming. 2. I am concerned about the generalization of PAT, specifically whether the optimized prompt can defend against adaptive and real-world attacks. 3. The robustness of the experimental results is questionable, particularly due to the use of prefix-based methods to calculate the ASR. These methods can easily lead to false negative or false positive judgments. 4. It is unnecessary for defenders to discuss their own capabilities. Generally, in most defense settings, defenders have full access to the systems (including models and data), as they are developed by the systems themselves.

Questions

1.The experimental settings are unclear; specifically, the number of training and testing datasets used in your setup is not specified. 2.Regarding general capabilities, why do the authors only test on MT-bench? Why not evaluate more general abilities such as math, coding, reasoning, etc.? 3.It is peculiar that compared to no defense, the performance on MT-bench is worse than the authors' proposed method. Could the authors explain this discrepancy?

Rating

4

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

1.The defense may be less effective when attackers implement other adaptive strategies (In the wild attack [2], PAP[3], GPTfuzzer[4], PAIR[5] etc.) with knowledge of the defense mechanism. 2. The rapid development of new jailbreak methods means that continuous updates to the defense strategy may be required. [2] "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. [3] Tree of Attacks: Jailbreaking Black-Box LLMs Automatically [4] GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts. [5] Jailbreaking Black Box Large Language Models in Twenty Queries

Summary

This paper presents a prompt adversarial tuning (PAT) to protect large language models against jailbreaking attacks. PAT is a prompt tuning-based defense for jailbreaking attacks where a string and GCG attack string are jointly optimized to have an LLM generate benign outputs. Experiments on AdvBench and MT-Bench show the defense is effective against GCG, AutoDAN, and ICA while maintaining MT-Bench performance.

Strengths

* The optimization objective of PAT is sound and incorporates the GCG attack as an adversary, which introduces the use of adversarial training techniques to support jailbreaking defenses * The experimental results show PAT generally outperforms previous defenses * Subject matter is important and timely

Weaknesses

* The evaluation only considers three defense baselines (ICD, self-reminder, and perplexity filtering) and three attack baselines (GCG, AutoDAN, and ICA). This does not cover many other attacks and defenses, such as SmoothLLM [1] and RPO [2] (which also has a similar methodology to PAT). * Closed-source model evaluation is also not comprehensive. Table 2 is missing the defense baselines, and Table 3 is missing both the defense baselines and the ICA attack. * The adaptive attack setting (4.5) only considers an adaptive GCG attack. This is unconvincing, as the defense string was optimized on GCG, so I wonder if adaptive versions of other attacks will transfer. If there aren't adaptive versions of the other attack baselines, the authors should detail this. [1] Robey, A., Wong, E., Hassani, H., & Pappas, G.J. (2023). SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks. ArXiv, abs/2310.03684. [2] Zhou, A., Li, B., & Wang, H. (2024). Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks. ArXiv, abs/2401.17263.

Questions

* The generated suffixes appear to have high perplexity. I wonder about the qualitative effect on model behavior for benign use, are there examples of this? * Why does the PAT string improve MT-bench scores in Table 1 but decrease scores in Table 3? * The threat model description suggests a gray-box setting (L96), but 4.1 claims to be about a white-box setting. Does this mean the adversary knows the defensive string?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

Authors discuss potential new adaptive attacks that can break the defense

Summary

This paper presents an in-context defense method against jailbreaking attacks against LLMs. The core idea is to tune the suffix to the system of a LLM with a multi-objective optimization framework: 1/ Benign prompts enhanced with this tuned suffix should activate normal response as if there were no suffix. 2/ Malicious prompts enhanced with the suffix should produce aligned answer to the user-specified benign response. The result shows the defense of the proposed method against GCG, AutoDAN and ICA under the white-box scenario. The empirical evaluation also cover the transferable defense effects over GPT-4.

Strengths

The idea of using max-min robust optimisation to adversarially tune the system prompt is well defined and explained clearly in this study. As well, adding suffix to the system prompt can provide a transferable defense across different model architectures of LLM.

Weaknesses

My major concern is this work does not extend comprehensively the coverage over attack and defense baselines in the empirical evaluation. I would suggest a few more attack and defense methods to enrich the current study: **Attack**: [1] Xie et al, Defending chatgpt against jailbreak attack via self-reminders, Nature Machine Intelligence, 2023. [2] Guo et al, COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability, ICML 2024. **Defense**: [3] Chao et al, Jailbreaking Black Box Large Language Models in Twenty Queries, Workshop on robustness of zero/few-shot learning in foundation models, NeurIPS 2023, https://arxiv.org/abs/2310.08419. [4] Xu et al, SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding, ACL 2024, https://arxiv.org/abs/2402.08983. As well, the evaluation needs to be extended to more open-sourced models, e.g. Mistral-8B and Llama-3 8B. Furthermore, I would be interested to check the transferability of the proposed defense method across different open-sourced models.

Questions

Please check the suggested additional empirical evaluation and discussion in the weakness.

Rating

4

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

This work still needs to extend its empirical evaluation to cover more attack / defense baselines and include more LLM models.

Summary

Inspired by prompt tuning and adversarial training, this paper proposed a new jailbreaking defense method named prompt adversarial tuning (PAT) which optimizes a defensive prefix by alternating between updating attack and defense controls with two opposite output targets.

Strengths

1. This paper employs adversarial training to defend against jailbreaking attacks, but instead of tuning model parameters, it tunes a defense prefix. Since the model parameters remain unchanged, the impact on utility is likely minimized. 2. The optimized defense prefix exhibits a certain degree of transferability across different LLMs.

Weaknesses

1. Another paper [1] proposed using an optimized soft system prompt to enhance model safety, which is highly relevant to this study. A corresponding comparison and discussion might be necessary. 2. For chat LLMs, the evaluation of utility should include both instruction-following ability and knowledge-wise capability. The authors only tested instruction-following ability on MT-bench, neglecting the knowledge-wise component. Therefore, additional utility experiments, such as those on MMLU, are necessary. [1] Zheng, Chujie, et al. "On prompt-driven safeguarding for large language models." Forty-first International Conference on Machine Learning. 2024.

Questions

The authors assume that the adversarial prompt from GCG is a suffix. However, in practice, an adversarial prompt can be inserted at any position. Therefore, my concern is whether the optimized defense prefix will become ineffective if the attacker changes the position of the adversarial prompt.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

Yes

Summary

This paper focuses on defense against jailbreak attacks. More specifically, it first considers improving model robustness through prompt tuning. The proposed method, Prompt Adversarial Tuning (PAT), aims to design a prefix to input prompts that encourages LLMs to still provide correct responses for benign inputs while responding with messages like "I am sorry, ..." to harmful prompts. To achieve this goal, the authors utilize prior attack methods to generate harmful prompts and then optimize the prefix to render these harmful prompts ineffective. Experiments on various datasets against different attack methods show the effectiveness of the proposal.

Strengths

1. The proposed method is effective and easy to implement. Importantly, it maintains benign utility, which is crucial. 2. The authors validated that the proposed method can transfer to different models, demonstrating its generalizability. 3. The method was also tested against adaptive attacks and proved effective.

Weaknesses

1. The proposed defense relies on obtaining harmful prompts through existing attack methods. This raises doubts about whether the proposed method can be effective against unseen jailbreak attacks. For instance, [b] found jailbreak attacks can be conducted using non-English prompts. There is not enough discussion on how well the proposed defense method works against unknown attacks [a]MULTILINGUAL JAILBREAK CHALLENGES IN LARGE LANGUAGE MODELS

Questions

1. To what extent is the proposed method effective against unseen jailbreak attacks?

Rating

6

Confidence

3

Soundness

4

Presentation

3

Contribution

3

Limitations

None

Acknowledgement

I thank the authors for their explanation, and I would like to keep my score.