Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters

Summary

In their paper, the authors introduce JAMBench, a benchmark designed to assess OpenAI's moderation guardrails using malicious questions. JAMBench includes 160 manually crafted questions across hate, sexual content, violence, and self-harm categories, categorized into medium and high severity levels. They also propose JAM (Jailbreak Against Moderation), a novel method employing cipher characters to bypass these guardrails by manipulating harm scores. The study demonstrates JAM's efficacy across various large language models (LLMs) and suggests potential countermeasures to strengthen moderation strategies, aiming to improve the safety of LLM-powered applications.

Strengths

1. This paper is both interesting and important. It reveals that even with input and output moderation defense, it is still possible to jailbreak LLMs for misuse. This finding is critical for improving the safety of closed-source LLMs such as ChatGPT. 2. The methods proposed for jailbreak and defense are reasonable. 3. The experimental results are solid and promising. 4. The paper writing quality is good.

Weaknesses

1. It would be interesting if the impact of the moderation models (for example, their capability and relation with the LLMs to be protected) can be analyzed and experimented. 2. The format seems a little tight such as the tables, probably due to the rich content and limit of space.

Questions

None

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

None

Summary

This work explores an interesting problem. Although the jailbreak prompt template can induce the LLM itself to generate harmful content, the malicious instruction and harmful content will be filtered out by moderation guardrails (such as input and output detection). How to jailbreak moderation guardrails has not been fully explored. In order to explore this problem, this work builds a benchmark called JAMBench and proposes a new jailbreak attack method called JAM.

Strengths

1. The problem this work focuses on is interesting. When conducting a jailbreak attack, one must consider not only how to jailbreak the LLM's built-in safety mechanism, but also how to jailbreak moderation guardrails (such as input detection and output detection). 2. The motivation of this work is reasonable. By collecting the moderation guardrail behaviors of closed-source LLMs, a small model (such as toxic-bert) is trained to guide the optimization of the inserted cipher characters. 3. Jailbreak input and output detection are two goals, and it is difficult to optimize inserted cipher characters to achieve both goals. This work derives a solvable space to simultaneously achieve the two goals.

Weaknesses

1. Is the construction of JAMbench reasonable and necessary? This work should guide experiments on adversarial samples that can successfully jailbreak the built-in safety mechanism of LLMs but fail to jailbreak moderation guardrails. However, the author claims that they only manually annotated test samples, which does not reflect the necessity of building JAMbench. Besides, since different LLMs have different safety performances, in order to explore the problem you raised, should you build test samples according to the safety performance of different LLMs instead of unified test samples? Therefore, it is difficult for me to believe in the rationality and necessity of JAMbench construction. 2. The optimization of cipher characters needs to be guided by the training samples. I am worried whether the training samples used in the optimization process and the final test samples are the same. If so, it is not uncommon to achieve a high success rate. You need to verify the generalization of your method. Although you provide experiments on other benchmarks, I observe that your improvements on other benchmarks are significantly lower than those on your constructed benchmark. Your jailbreak template combines many other tricks, such as attack prefixes and the carefully designed of jailbreak prompt templates. You need to prove that the improvement is brought by your optimized cipher characters, not other factors. As you demonstrate in your ablation experiments, the attack prefix is necessary, but this is not the contribution of your work.

Questions

Please refer to the weaknesses. If you can convince me, I will consider changing my score.

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

None

Summary

In this paper, the authors proposed a jailbreak bench and a corresponding method to bypass the moderation guardrail of LLMs. The proposed JAMBench is proved to be effective in triggering the filtered-out error by the moderation guardrail of LLMs. Besides, the proposed JAM method is effective in bypassing moderation guardrails.

Strengths

1. This paper introduced a new jailbreak bench specified for moderation guardrail of LLMs. The JAMBench can trigger the filtered-out error by the moderation guardrail of LLMs. 2. A jailbreak method named JAM which is effective in bypassing the moderation guardrail of LLMs.

Weaknesses

1. The perplexity of JAM is significantly larger than many baselines such as ICA, PAIR and CipherChat, which makes it easier to leverage perplexity filter to filter the prompts.

Questions

1. How to ensure the diversity of the prompts in each category of JAMBench? The diversity of prompts is important for evaluating the moderation guardrail of LLMs.

Rating

5

Confidence

5

Soundness

3

Presentation

2

Contribution

2

Limitations

1. For the shadow model, the evaluation method or discussion regarding ensuring the shadow can mimic the moderation guardrail is missing in the paper. 2. The proposed Cipher Character optimization shares some similar motivation as the universal prompt optimization. The comparison between these two methods is missing in the paper.