Library

When LLM Meets DRL: Advancing Jailbreaking Efficiency via DRL-guided Search

Recent studies developed jailbreaking attacks, which construct jailbreaking prompts to fool LLMs into responding to harmful questions. Early-stage jailbreaking attacks require access to model internals or significant human efforts. More advanced attacks utilize genetic algorithms for automatic and black-box attacks. However, the random nature of genetic algorithms significantly limits the effectiveness of these attacks. In this paper, we propose RLbreaker, a black-box jailbreaking attack driven by deep reinforcement learning (DRL). We model jailbreaking as a search problem and design an RL agent to guide the search, which is more effective and has less randomness than stochastic search, such as genetic algorithms. Specifically, we design a customized DRL system for the jailbreaking problem, including a novel reward function and a customized proximal policy optimization (PPO) algorithm. Through extensive experiments, we demonstrate that RLbreaker is much more effective than existing jailbreaking attacks against six state-of-the-art (SOTA) LLMs. We also show that RLbreaker is robust against three SOTA defenses and its trained agents can transfer across different LLMs. We further validate the key design choices of RLbreaker via a comprehensive ablation study.

cs.CRComputer Science
Source

Paper

References (74)

01PathSeeker: Exploring LLM Security Vulnerabilities with a Reinforcement Learning-Based Jailbreak ApproachZhihao Lin, Wei Ma, Mi Zhou et al.2024 · arXiv.org · 17 citations In Library
02Bileve: Securing Text Provenance in Large Language Models Against Spoofing with Bi-level SignatureTong Zhou, Xuandong Zhao, Xiaolin Xu et al.2024 · Neural Information Processing Systems · 25 citations In Library
03TrojFM: Resource-efficient Backdoor Attacks against Very Large Foundation ModelsYuzhou Nie, Yanting Wang, Jinyuan Jia et al.2024 · arXiv.org · 9 citations In Library
04On Large Language Models’ Resilience to Coercive InterrogationZhuo Zhang, Guangyu Shen, Guanhong Tao et al.2024 · IEEE Symposium on Security and Privacy · 25 citations
05MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual TokensKirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman et al.2024 · arXiv.org · 149 citations In Library
06RigorLLM: Resilient Guardrails for Large Language Models against Undesired ContentZhuowen Yuan, Zidi Xiong, Yi Zeng et al.2024 · International Conference on Machine Learning · 85 citations In Library
07AutoDefense: Multi-Agent LLM Defense against Jailbreak AttacksYifan Zeng, Yiran Wu, Xiao Zhang et al.2024 · arXiv.org · 170 citations In Library
08Curiosity-driven Red-teaming for Large Language ModelsZhang-Wei Hong, Idan Shenfeld, Tsun-Hsuan Wang et al.2024 · International Conference on Learning Representations · 93 citations In Library
09SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware DecodingZhangchen Xu, Fengqing Jiang, Luyao Niu et al.2024 · Annual Meeting of the Association for Computational Linguistics · 282 citations In Library
10Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit CluesZhiyuan Chang, Mingyang Li, Yi Liu et al.2024 · Annual Meeting of the Association for Computational Linguistics · 83 citations In Library
11COLD-Attack: Jailbreaking LLMs with Stealthiness and ControllabilityXing-ming Guo, Fangxu Yu, Huan Zhang et al.2024 · International Conference on Machine Learning · 195 citations In Library
12Rapid Optimization for Jailbreaking LLMs via Subconscious Exploitation and EchopraxiaGuangyu Shen, Siyuan Cheng, Kai-xian Zhang et al.2024 · arXiv.org · 19 citations In Library

Scroll for more · 38 remaining

Similar papers

RL-JACK: Reinforcement Learning-powered Black-box Jailbreaking Attack against LLMsXuan Chen, Yuzhou Nie, Lu Yan et al.2024 · arXiv.org · 32 citations In Library
xJailbreak: Representation Space Guided Reinforcement Learning for Interpretable LLM JailbreakingSunbowen Lee, Shiwen Ni, Chi Wei et al.2025 · arXiv.org · 12 citations In Library
A Systematic Investigation of RL-Jailbreaking in LLMsMontaser Mohammedalamen, Kevin Roice, Reginald McLean et al.2026 · 0 citations In Library
TrailBlazer: History-Guided Reinforcement Learning for Black-Box LLM JailbreakingSunghoon Yoon, Ruizhi Qian, Minda Zhao et al.2026 · arXiv.org · 1 citation In Library
Jailbreak-R1: Exploring the Jailbreak Capabilities of LLMs via Reinforcement LearningWeiyang Guo, Zesheng Shi, Zhuo Li et al.2025 · arXiv.org · 17 citations In Library
"To Survive, I Must Defect": Jailbreaking LLMs via the Game-Theory ScenariosZhen Sun, Zongmin Zhang, Deqi Liang et al.2025 · arXiv.org · 4 citations In Library
SRTJ: Self-Evolving Rule-Driven Training-Free LLM JailbreakingJindong Li, Ying Liu, Yali Fu et al.2026 · arXiv.org · 0 citations In Library
Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive AttacksMaksym Andriushchenko, Francesco Croce, Nicolas Flammarion2024 · International Conference on Learning Representations · 544 citations In Library
PathSeeker: Exploring LLM Security Vulnerabilities with a Reinforcement Learning-Based Jailbreak ApproachZhihao Lin, Wei Ma, Mi Zhou et al.2024 · arXiv.org · 17 citations In Library
Formalization Driven LLM Prompt Jailbreaking via Reinforcement LearningZhaoqi Wang, Daqing He, Zijian Zhang et al.2025 · arXiv.org · 4 citations In Library

Peer review

Accept (poster)NeurIPS 2024 · via OpenReview
Reviewer 5U1Z6/10 · confidence 4/52024-06-25

Summary

This paper proposed a black-box jailbreaking framework, RLbreaker, it use Reinforcement learning to help the optimization of jailbreaking prompt. At training each step, the designed agent of RLbreaker will select a mutator from a small set, and then the helper LLM will use the selected mutator to enhance the prompt. The optimized prompt will be feed to victim LLM and an un-aligned LLM, the framework will use the difference between 2 responses to calculate the reward for next training step. The experiment section include general attack and transfer attack, as well as attack against Jailbreaking Defenses.

Strengths

Compare with the most popular baselines (i.e. the GCG and AutoDAN), the framework proposed by this paper decrease the search space so that the training speed is fast. The structure of the paper is very clear and it is easy to understand the framework. The section 4, consists of 3 kinds of evaluation (general/transfer/ablation), is reasonable. Although the action base is small, but in the ablation section, the results show that randomly select action will lead a significant performance drop. This proves that the agent is definitely effective.

Weaknesses

All experiments are done on a 300-samples test set, and it is the same distribution as the training set. Therefore hard to prove its robustness and effectiveness.

Questions

How did you determine the action space you ultimately chose?

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

Yes

Reviewer 5ZZd6/10 · confidence 3/52024-07-13

Summary

This work proposes a new method to jailbreak LLM to elicit harmful respones. It adapts deep reinforcement learning to learn a policy of sampling jailbreaking operations (modifying prompt) from a predefined pool. A LLM is then used to rewrite the query prompt complying with the sampled operation. To guide the policy learning, it proposes a new reward function that compares the difference between the target model's response and a reference response. The empirical results confirm the efficacy and the efficiency of the proposed method.

Strengths

1. the studied problem, jailbreaking LLM, is an important topic of AI safety. 2. although applying DRL to text optimization is not novel, the way how different components are designed and put together in this work is great and interesting. 3. the efficacy improvement of the proposed method is large in some cases but not all.

Weaknesses

1. For black-box evaluation, GPT-3.5-turbo may be dated. GPT-4 is recommeded. 2. I am a little concerned about the results of GPT-Judge since it shows an inconsistent scale compared to Sim. For example, for Llama2-70b-chat in Tab. 1, the performance improvement of the proposed method indicated by GPT-Judge over the previous methods is dramatically larger than that indicated by Sim. The similar cases also occur in Tab. 2 an Tab. 3. 3. The training of RL agent may be instable. How many runs of training did authors test to report the results e.g. in Tab.1. How much is the variance? 4. For the results under Sim., the proposed method sometimes underperform the previous works, e.g., on Mixtral in Tab.1. Similar situations also occur in Tab.2 and Tab.3. Do authors have some insights about why? 5. The efficiency the proposed method is a concern. As shown in Tab. 8, the proposed method, even though more efficient than AutoDAN and GCG, costs much more than other blackbox methods on many models.

Questions

see the Weakness above.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

limitations are discussed.

Reviewer ipyX5/10 · confidence 4/52024-07-13

Summary

This paper proposes a new jailbreaking attack on LLMs with deep-reinforcement learning (DRL) techniques, which takes jailbreak prompts as states and mutations as actions.

Strengths

1. This paper is the first to leverage DRL techniques to jailbreaking LLMs, bringing new insights to this community. 2. The experiments include multiple LLMs and attack baselines. 3. The evaluation considers ablation studies on each part of the proposed method, showing the robustness against hyper-parameters of the DRL training.

Weaknesses

1. The organization of this paper can be substantially improved to polish readability. For example, Section 2 failed to discuss the background (deep) reinforcement learning and their related work on jailbreaking. 2. Some technique details of the method are not specified. For example, in line 164, which mutators are used are not clearly introduced. Instead, the authors simply use a reference here. 3. The plausibleness of the reference answer $\hat u_i$ derived from an unaligned LLM should be further justified. Specifically, the weakly aligned vicuna-13b does not always respond to harmful prompts. What happens if both the target LLM and this model refuse to answer a harmful query? 4. The experiment does not indicate the training/inference time required for the method. 5. The experiment results show that the improvement of the proposed method over existing methods is somewhat limited.

Questions

See weaknesses.

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

N/A

Reviewer ipyX2024-08-13

Thanks for the rebuttal, your efforts are truly appreciated. I've raised my rating to 5. Some comments: - I know that there are no existing works that specifically apply DRL to jailbreaking attacks (see strengths); what I requested here is to discuss how (deep) reinforcement learning has been used to attack ML models / LLMs. Such connections can help the readers better understand to what extent these techniques have been leveraged in the adversarial ML area. - The limitation on the requirement of manually double-checking all the reference answers is a limitation that needs to be explicitly acknowledged. - It would be great to evaluate stronger attack/defense baselines, like the demonstration-based in-context attack/defense (https://arxiv.org/abs/2310.06387).

Authorsrebuttal2024-08-13

Thanks to the Reviewer ipyX for their insightful comments. We really appreciate the reviewer for raising their score to acknowledge our effort. Below, we would like to provide some clarifications for the reviewer's additional comments. 1. We totally agree with the reviewer that discussing how DRL has been used in attacking LLMs if not for jailbreaking purposes is super helpful. We have a short summary of existing works in Section 2. We also experiment with a baseline (token-level RL) that is generalized from existing works in our evaluation. We will further emphasize these in our paper. 2. We would like to clarify that we did a manual check just to assess the quality of the unaligned models in generating reference questions. This is a one-time effort. It is like an ablation study or sanity check and is not required to run our method. 3. Moving forward, we will follow the reviewer's suggestion to add new baselines, including the one pointed out by the reviewer.

Reviewer pYsF6/10 · confidence 3/52024-07-22

Summary

This paper introduces RLbreaker, a novel deep reinforcement learning (DRL) approach for generating jailbreaking prompts to attack large language models (LLMs). The authors frame jailbreaking as a search problem and design a DRL agent to guide the search process more efficiently than existing stochastic methods. Key technical contributions include a customized reward function, action space based on prompt mutators, and a modified proximal policy optimization (PPO) algorithm. Through extensive experiments, the authors demonstrate that RLbreaker outperforms state-of-the-art jailbreaking attacks across multiple LLMs, shows resilience against existing defenses, and exhibits strong transferability across models. The paper also includes ablation studies validating the key design choices.

Strengths

- Good results on Llama-2-70B (52.5% ASR). - The proposed method makes sense, and, to the best of my knowledge, no one has proposed using an RL algorithm for guiding jailbreak search. - The transferability of the attack is non-trivial.

Weaknesses

I have concerns about the reported numbers: - Why is GCG shown as N/A for GPT-3.5 Turbo? Evaluating the result of a transfer attack would still make sense (and I would expect that the ASR according to GPT-4 as a judge should be above 50%). - Moreover, why does PAIR perform so badly on GPT-3.5 Turbo? You report 9% ASR while the original paper reports 60% ASR (but on a different set of AdvBench requests). Similarly, Mixtral is known to be a non-robust model, so it’s surprising to see that PAIR achieves only 15% ASR on it. - Why does the ASR according to the GPT judge is significantly higher when the perplexity filter is enabled (Table 2: 69.1%, Table 1: 52.5%)? Other concerns: - Minor: “However, the random nature of genetic algorithms significantly limits the effectiveness of these attacks.” - why? - Minor: Tables 2 and 3 have a too small font. I'd be willing to increase the score if the concerns about the reported numbers are resolved.

Questions

See the questions about the reported numbers.

Rating

6

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

Yes.

Reviewer 5U1Z2024-08-11

Thanks for your rebuttal. I will keep the Rating since it is already the highest.

Authorsrebuttal2024-08-12

We thank the reviewer for supporting the paper and maintaining the highest rating. We will update our paper based on the suggestions.

Reviewer 5ZZd2024-08-11

Thanks much for your thorough responses. Most of my concerns have been addressed, so I decide to raise my score to 6. However, I still have concern about the faithfulness of evaluation metric, GPT-Judge. I understand that this is a previously accepted practice. I personally question it because of the existing observation on the instability of LLM in answering the question when it is written in different prompts of the same meaning like [1]. [1] Zong et al., Fool Your (Vision and) Language Model With Embarrassingly Simple Permutations, ICML 2024.

Authorsrebuttal2024-08-12

We sincerely appreciate the reviewer for updating the score! We are happy that our response could help address the reviewer's concern. Regarding the reliability of the GPT-Judge metrics, we agree with the reviewer that it is not an entirely stable metric for assessing the success of a jailbreaking attack. However, as shown in both [1] and [2], GPT-4 demonstrates a high correlation with human judgment (0.9), suggesting its potential utility as a reliable verifier for responses provided by victim LLMs. While human annotation remains a more accurate method of judgment, it is costly given our experimental setup, which requires evaluating 320 questions in the test set for 5 baselines and our method. Therefore, we believe GPT-Judge serves as a cost-effective and efficient alternative for evaluating attack effectiveness. Furthermore, we ensure consistency in our evaluations by applying the same metrics across all baseline and our method. In our future work, we will carefully consider the reviewer's suggestions and explore more strategies to enhance the reliability and stability of our judgment metrics. For example, integrating results from a harmful content classifier [3] alongside GPT-Judge, or grouping evaluations from multiple LLMs and perform a majority vote. [1] COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability. Guo et al., ICML 2024. [2] AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models. Liu et al., ICLR 2024. [3] HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. Mazeika et al., arXiv 2024.

Reviewer pYsF2024-08-12

Follow-up comment

Thanks for the detailed response, it addresses my concerns. I increase my score to 6.

Authorsrebuttal2024-08-12

We would like to thank the reviewer for increasing the score and we are happy that our response could help address the reviewer's concern. As we proceed with the revision, we will be mindful of the suggestions and present a stronger version of our paper based on the reviewer's feedback.

Authorsrebuttal2024-08-13

Thanks to the Reviewer ipyX again for the insightful comments and questions. Since the discussion phase is about to end, we are writing to kindly ask if the reviewer has any additional comments regarding our response. We are at their disposal for any further questions. In addition, if our new experiments address the reviewer's concern, we would like to kindly ask if the reviewer could reconsider their score.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC