Library

Efficient Adversarial Training in LLMs with Continuous Attacks

Large language models (LLMs) are vulnerable to adversarial attacks that can bypass their safety guardrails. In many domains, adversarial training has proven to be one of the most promising methods to reliably improve robustness against such attacks. Yet, in the context of LLMs, current methods for adversarial training are hindered by the high computational costs required to perform discrete adversarial attacks at each training iteration. We address this problem by instead calculating adversarial attacks in the continuous embedding space of the LLM, which is orders of magnitudes more efficient. We propose a fast adversarial training algorithm (C-AdvUL) composed of two losses: the first makes the model robust on continuous embedding attacks computed on an adversarial behaviour dataset; the second ensures the usefulness of the final model by fine-tuning on utility data. Moreover, we introduce C-AdvIPO, an adversarial variant of IPO that does not require utility data for adversarially robust alignment. Our empirical evaluation on five models from different families (Gemma, Phi3, Mistral, Zephyr, Llama2) and at different scales (2B, 3.8B, 7B) shows that both algorithms substantially enhance LLM robustness against discrete attacks (GCG, AutoDAN, PAIR), while maintaining utility. Our results demonstrate that robustness to continuous perturbations can extrapolate to discrete threat models. Thereby, we present a path toward scalable adversarial training algorithms for robustly aligning LLMs.

cs.LGcs.CRAdversarial Robustness in Machine LearningComputer Science
Source

Paper

Similar papers

Information Theoretic Adversarial Training of Large Language ModelsYiwei Zhang, Jeremiah Birrell, Reza Ebrahimi et al.2026 · arXiv.org · 0 citations In Library
Understanding and Improving Continuous Adversarial Training for LLMs via In-context Learning TheoryShaopeng Fu, Di Wang2026 · arXiv.org · 1 citation In Library
Adversarial Attacks on Large Language Models Using Regularized RelaxationSamuel Jacob Chacko, Sajib Biswas, Chashi Mahiul Islam et al.2024 · arXiv (Cornell University) · 17 citations In Library
MixAT: Combining Continuous and Discrete Adversarial Training for LLMsCsaba D'ek'any, Stefan Balauca, Robin Staab et al.2025 · Neural Information Processing Systems · 11 citations In Library
AdvAttack-LLM: Exploiting Elasticity in Large Language Models for Efficient Adversarial AttacksLijia Yu, Yan Cao2025 In Library
DarkLLM: Learning Language-Driven Adversarial Attacks with Large Language ModelsYerui Sun, Xin Wang, Jiaming Zhang et al.2026 · arXiv.org · 0 citations In Library
Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMsA. Sheshadri, Aidan Ewart, Phillip Guo et al.2024 · Trans. Mach. Learn. Res. · 151 citations In Library
AntiDote: Bi-level Adversarial Training for Tamper-Resistant LLMsDebdeep Sanyal, Manodeep Ray, Murari Mandal2026 · Proceedings of the AAAI Conference on Artificial Intelligence In Library
AntiDote: Bi-level Adversarial Training for Tamper-Resistant LLMsDebdeep Sanyal, Manodeep Ray, Murari Mandal2025 · AAAI Conference on Artificial Intelligence · 3 citations In Library
Robust LLM safeguarding via refusal feature adversarial trainingLei Yu, Virginie Do, Karen Hambardzumyan et al.2024 · arXiv (Cornell University) · 69 citations In Library

Peer review

Accept (spotlight)NeurIPS 2024 · via OpenReview
Reviewer Z2aE5/10 · confidence 3/52024-06-21

Summary

Large language models (LLMs) are vulnerable to adversarial attacks that can bypass their safety guardrails. However, current adversarial training methods for LLMs are hindered by the high computational costs required to perform discrete adversarial attacks at each training iteration. To solve this problem, the paper instead calculates adversarial attacks in the continuous embedding space of the LLM, which is orders of magnitudes more efficient.

Strengths

The presentation of the article is clear. The proposed method can consume less computational cost to obtain robust LLMs against adversarial attacks.

Weaknesses

1. The main claim of the paper is that the proposed method can use less computational cost to obtain a robust LLM, and regarding this viewpoint, there is only an experimental setting presented in Table 1, which may be insufficient. I believe that some more intuitive experiments should be added, such as comparative experiments on the real-time cost for a single example and for the entire process. 2. Although the paper is overall clear and the problem it intends to solve is well-defined, some details are not clear enough, especially the description of the proposed method's pipeline. For example, 'IPO' keeps appearing in the first two pages, but only in Section 3 are some descriptions of 'IPO' provided. 3. The hardware description in Section 4 is also unclear. I understand that this does not impact the main contributions of the paper, but a more explicit explanation of each experiment's settings would provide a clearer understanding for the readers. Additionally, totaling GPU hours across different GPUs may be unreasonable, as 1904 GPU hours on a V100 differ from those on 80GB A100 GPUs. Therefore, I hope the authors can further refine these details. 4. As LLAMA-2 is one of the most mainstream open-source LLMs, why have the experiments yet to be conducted on LLAMA-2 to evaluate the effectiveness of the proposed method? 5. Currently, the defense mechanisms on LLMs are not limited to R2D2 nor even to AT methods. I'm curious about how the proposed method compares with other methods, such as some methods mentioned in the related works.

Questions

See Weaknesses.

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors have discussed limitations of the work.

Reviewer TJPv5/10 · confidence 3/52024-07-07

Summary

This paper introduces adversarial attacks in continuous space in the context of LLMs. In addition, it utilizes continuous attacks for adversarial training to robustify LLMs and demonstrate that this efficient training algorithm can indeed protect LLMs against various attacks, including discrete ones, while maintaining the utility.

Strengths

1. The proposed method in this paper is easy to understand, to implementation and generally applicable. 2. The proposed method dramatically improve the efficiency of adversarial training in the context of LLMs. 3. The proposed method does not hurt the utility of the model too much.

Weaknesses

1. The experiments should be more comprehensive: (1) In Figure 2, the comparison between R2D2 and the proposed methods is only conducted on the model ZEPHYR-7B, comparisons on more models would be preferred. (2) The authors only test one safe answer ''Sorry, I can't do that''. The safe answer $y$ is a very important variable in the loss objective function. Tests on more ''safe answers'' would make the results more convincing and more generally applicable. (3) The attack success rates are based on GCG, AutoDAN and PAIR. Recently, there have been some stronger attacks proposed, such as LLM-adaptive attacks (https://github.com/tml-epfl/llm-adaptive-attacks), the authors should include more attacks for a more comprehensive evaluation. (4) Ablation studies should be conducted to support why IPO is better than DPO, as line 161 to 162 claimed. 2. The proposed methods introduce more hyper-parameters to tune, hindering its application for practitioners. 3. The presentation of the paper can be improved, for example, the formulation in line 154 to 156 is very confusing, as $\mathcal{L}'$ is not defined.

Questions

The major concerns have been pointed out in the weakness part, the authors should first answer questions in that section. In addition, we have some minor questions: 1. In line 210 to 211, why do bigger models have smaller values of $\epsilon$? Is there any intuition? 2. Similar to TRADES [Zhang et. al 2019], in addition to the value of $\epsilon$, is it possible to add a co-efficient to the last term of Equation (4) to balance the trade-offs between robustness and utility? Overall, the paper tackles an interesting problem. However, due to the weakness part and the questions above, I believe the manuscript needs to be edited to improve. I welcome the authors to address my concerns during the rebuttal and will re-evaluate the manuscript after the rebuttal. > Post Rebuttal I improved my ratings to 5 after reading authors' rebuttal.

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

3

Limitations

The limitations and the societal impact are discussed at the end of the paper.

Authorsrebuttal2024-08-13

Thank you for your reply and feedback. We chose to evaluate our method with a subset of the strongest attack available at the point of submission (according to Harmbench [1]). Beyond the ICL attack and the adaptive attack that we performed for the rebuttal, we will conduct additional evaluations with the approaches proposed in [2, 3] for the final manuscript. These attacks have shown high ASR against robust models. We will add our ablation study on the behavior of $\epsilon$ for models of different sizes to the final manuscript. We are open to any further suggestions on ablation studies that would further improve our work. We will add a more profound discussion about the differences between previous approaches in encoder-decoder models and the presented adversarial training algorithm. More specifically, we will: - Elaborate on the different training settings, i.e., post-training stage for LLMs and pre-training in encoder-decoder models - Differences in the optimization target, i.e., robustness in classification settings vs. preference optimization or alignment finetuning - Differences in the optimization goal, i.e., improved generalization vs. adversarial robust alignment. We want to emphasize that robustness in classification problems differs substantially from adversarial robust alignment. In the classification setting, models are trained to be robust to minor input perturbations with respect to their prediction. In contrast, in the alignment setting, we are not concerned with the stability of predictions. Rather, we train the model to refrain from generating "toxic" outputs altogether. To achieve this, losses need to be adjusted for this task (such as combining different objectives or using preference optimization algorithms). None of the previous methods was designed for LLM alignment and they can not be employed for this task without further changes. Lastly, the majority of these approaches were used during the pre-training stage of encoder-decoder models. Applying these algorithms during the pre-training stage of an LLM exceeds our computational budget by orders of magnitude. [1] Zou et al., "HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal" (Feb. 2024) [2] Thompson T and Sklar M., "Fluent Student-Teacher Redteaming" (July 2024) [3] Liao et al., "AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs" (May 2024)

Reviewer Jk9f7/10 · confidence 4/52024-07-08

Summary

This paper proposes adversarial training for LLMs, in which perturbations are created in the continuous embedding space rather than finding discrete suffixes. The proposed fast adversarial training algorithm (AdvUL) consists of two losses: the first strengthens the model against continuous embedding attacks computed on an adversarial behavior dataset, and the second ensures the final model's usefulness by fine-tuning utility data. Furthermore, the authors introduce C-AdvIPO, an adversarial variant of IPO that does not rely on utility data for adversarially robust alignment. The empirical evaluation of four models, Gemma, Phi3, Mistral, and Zephyr, at various scales, reveals that both algorithms improve LLM robustness against discrete attacks (GCG, AutoDAN, PAIR).

Strengths

The Strengths of this paper include: - The writing of this paper is clear, with neat formulas to fastly explain the proposed AT methods. Using the form of DPO/IPO in Eq. (5) seems reasonable to me. - There are details to describe the experimental settings, and experiments are comprehensively done on different LLMs and datasets. - The empirical improvements on the robustness against GCG, AutoDAN, and PAIR appear significant.

Weaknesses

The Weaknesses of this paper include: - The main difference between adversarial attacks/defenses on traditional models (e.g., CNNs) and those on LLMs is that there is no explicitly defined *threat model* when jailbreaking LLMs. Namely, the strategy to jailbreak an LLM could be quite flexible, and there is usually no constraint on human imperceptibility. AT is a strong defense under a given threat model (e.g., $8/255, \\ell\_{\\infty}$), but AT is observed to generalize badly to unseen attacks. While I'm experienced with AT in traditional settings, I'm still not very convinced that AT could be a good solution for LLMs. - While GCG, AutoDAN, and PAIR are commonly evaluated attacks, they are relatively weak nowadays. The authors are encouraged to evaluate their models against more advanced and diverse attacks, such as ICL-based [1,2] and/or decomposing-based [3]. - To investigate the limits of C-AdvUL and C-AdvIPO, there should be a sanity check to directly perform attacks in the continuous embedding space. - From my perspective, what C-AdvUL and C-AdvIPO do is connected to machine unlearning (i.e., unlearning the harmful knowledge), where [4,5] have used similar learning objectives. Their connections should be more discussed and empirically ablated. Refs:\ [1] Many-Shot Jailbreaking\ [2] Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defenses\ [3] A False Sense of Safety: Unsafe Information Leakage in 'Safe' AI Responses\ [4] Negating Negatives: Alignment without Human Positive Samples via Distributional Dispreference Optimization\ [5] Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning

Questions

Why there are no experiments done on the Llama series of models? Is there any reason for this design choice?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Please see Weaknesses.

Authorsrebuttal2024-08-10

We thank the reviewer for their quick response and for increasing their score! We agree that further attack evaluations would improve our work and will look into adding decomposition attacks to our work. We welcome any further feedback to improve our paper and are thankful for your feedback.

Reviewer Jk9f2024-08-09

I thank the authors for their responses and additional experiments, which addressed the majority of my concerns. I understand that stronger attacks like [1,2,3] occur concurrently with the submission, so I didn't take this as a negative point. Nonetheless, I encourage the authors to conduct more comprehensive evaluations in their paper revision, particularly against attacks that differ from the (implicit) assumptions of C-AdvUL and C-AdvIPO. In conclusion, I believe this paper provides a good defense against jailbreaks through extensive experiments. So I'd like to raise my score to 7.

Reviewer Z2aE2024-08-11

Thanks for the responses. Most of my concerns have been addressed. However, I still have concerns about W5: The authors shared a series of similar works in the related work section, but failed to compare them in the experimental section. This can easily lead readers to question the accuracy of the limitations of related works. Of course, I agree with authors’ explanation that some methods are orthogonal or irreproducible. Nevertheless, this does not substantially solve the above problem, although I acknowledge your explanation. I hope the authors can handle this issue more appropriately in the future. Hence, I maintain my score.

Authorsrebuttal2024-08-11

Thank you for your feedback on our paper. We'd like to clarify our position: The primary contribution of our paper is to address a specific research question: - Does adversarial training with continuous attacks in the token embedding space of an LLM provide robustness to discrete natural language attacks? Prior works on adversarial training in NLP target different objectives, such as improved generalization or adversarial robustness in sentiment classification. These methods differ in several key aspects: Previous work didn't consider threat models, where the attacker has complete input control but generally enforced similarity to the initial text. This was done to keep the adversarial nature of the perturbation (to be semantically not meaningful to a human observer). However, we focus on alignment and not classification, which results in substantially different threat models. Further, not all methods directly apply to decoder-only LLMs, and some only apply to pretraining, demanding unrealistic amounts of compute in the LLM setting. We want to emphasize that the scope of our paper is focused first on answering our research question and second on adversarial robust alignment algorithms. We believe the prior work mentioned above cannot be considered as a baseline that should be compared to our algorithms, as extending such prior work to our setting would be a contribution by itself. Instead, we adopted two of the most common losses used in LLM training. However, we acknowledge that leveraging such prior work to improve robustness of alignment is an interesting avenue for future work, and we will discuss this option in the outlook section of the final manuscript.

Reviewer TJPv2024-08-13

I thank the authors for detailed feedbacks. Most of my concerns have been address, so I am willing to improve my rating to 5. The revised manuscript should include stronger attacks (adaptive attacks) and more ablation studies. I agree with Reviewer Z2aE that the author should highlight the difference of their proposed defence method and existing ones.

Program Chairsdecision2024-09-25

Decision

Accept (spotlight)

© 2026 NYSGPT2525 LLC