Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space

Summary

This paper demonstrates the attack vector using soft prompt tuning (prompt optimization in the token embedding space) for jailbreaking aligned LLMs and for “breaking” unlearned LLMs.

Strengths

### Significance I believe that the problem studied in this paper is well-motivated. Soft prompts are a threat that is specific to open-source models. However, as the authors mentioned, open-source models are becoming increasingly powerful and arguably can be utilized for harmful purposes (writing fake news, phishing, etc.). Soft prompting (along with other fine-tuning attacks) will continue to be a challenging problem to fix for open-source models. These threats call into question the trade-off between benefits and negative impacts open-source foundation models have on society. This work well demonstrates a potential negative consequence that cannot be easily mitigated by fine-tuning or alignment methods.

Weaknesses

### 1. Utility metrics L108: “Utility” is potentially not the right terminology or metric for an attack. I’m also unsure what its purpose is or why the attacker should care about utility of the attacked model. It would make sense for defenses. If “utility” here is intended to measure how much the model’s response helps the attacker in conducting a harmful task, I believe it should be part of the success rate metric, i.e., whether the model generates truly harmful responses. But my understanding is that this is not the authors’ intention. ### 2. Attack success metrics The authors mention five different metrics just for measuring an attack success rate. There should be better a motivation for why or whether we should consider all of them (vs just pick 1-2). It seems more reasonable to pick only one metric that best correlates with human judgement. I do not immediately see how each of these metrics add any relevant information beyond that, apart from the "cumulative success rate" (CU). It is also unclear whether the jailbreaking task and the unlearning task should rely on the same metric. I believe Section 4.3 can be better organized. ### 3. Toxicity score I would like to see some justification regarding using this model for detecting toxic strings vs approaches used by a few other works (GPT-4, LlamaGuard, LLM-based detectors from https://github.com/centerforaisafety/HarmBench?tab=readme-ov-file#-classifiers). I’m thinking that these classifiers may be more reliable than the toxic-bert model. ### 4. Cumulative success rate While I’m broadly in favor of the cumulative success rate or the “pass@k”-style metric, I don’t agree that increasing a one-shot attack to a k-shot one fully aligns with Kerckhoffs’s Principle. Kerckhoff’s Principle is about security by design and having a rigorous theoretical guarantee in a way that even when the attackers know every detail of the defense, they cannot break it. There should be a discussion on a reasonable choice of k (if one sets k to an extremely large number, sampling with high temperature would eventually lead to at least one bad response even if the model is not attacked). For the unlearning task, why does a k-shot attack matter? If the model only outputs the right answer 1 out of 100 times, how does the attacker pick out the right answer? A k-shot attack evaluation is more k times more expensive than 1 shot. So I’d like to be convinced why it matters, e.g., if the k-shot success rate is completely correlated with the 1-shot, should I care about the k-shot? ### 5. Perplexity I’m not convinced that perplexity is the right metric in Figure 4. Is it intended to measure the utility? The original and the attacked models would say something completely different so I doubt perplexity is a meaningful comparison here. Low perplexity can be achieved by repeating a single token. A few more questions: how is perplexity measured? Is it computed on standard, attacked, or a third model? I see that perplexity does not include prompt, but is it conditioned on the prompt? ### 6. Lack of baseline attacks While there are a lot of results and discussion on universal attacks and multiple metrics, the numbers reported for the baseline attacks only include ASR and computation cost. In the unlearning setting, there is also no baseline attack at all (e.g., discrete optimization or other prior attacks). ### 7. Novelty One of primary weaknesses of the paper is the lack of novelty. Soft prompt or prompt tuning is an idea that has been around for a few years and has been touched on in multiple papers in the past [1, 2, 3]. These papers, while not focusing on soft prompts as the main threat, propose attacks that optimize in the continuous space (embedding or vocabulary probability space) and then use different heuristics to recover hard prompts. Therefore, I believe that this paper offers limited technical contribution. The experiments in this paper are fairly extensive, i.e., covering a lot of interesting questions, but there is a lack of depth in how the questions are being explored scientifically. For example, I find the fact that universal attacks are better than individual attacks very interesting and under-explored (it is also observed in the GCG paper I believe). Prompting the embedding attacked model to generate jailbreaks for another model seems to also work surprisingly well. These are potentially more novel scientific questions that are buried in the paper. On the other hand, the main contribution of showing that the soft prompt attack works is rather expected and has limited contributions to the scientific community. 1. https://arxiv.org/abs/2104.13733 2. https://arxiv.org/abs/2302.03668 3. https://arxiv.org/abs/2401.16656

Questions

**Q1:** What dataset is used for the jailbreaking experiment? **Q2:** > An additional investigation confirms that, across all scenarios, attacks from the final optimization iteration were less successful than those from earlier iterations. For simplicity, we nevertheless report the success at the last iteration. Did the authors use any early stopping method here? **Q3:** On the unlearning task (Section 6) and jailbreak task (Figure 3), universal attacks appear to perform better than individual attacks. This is very surprising and worth addressing in my opinion. First of all, do I have the right understanding here? Do the authors have explanation for why a direct optimization (individual attack) is *worse* than optimizing on a number of different samples and essentially relying on generalization (universal attack)?

Rating

4

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

This has been addressed adequately.

Summary

The paper discusses a new adversarial attacking approach called "embedding space attacks" targeting open-source large language models (LLMs). Overall, traditional adversarial methods focus on discrete input manipulations at the token level, effective in closed-source environments accessed via APIs. However, with the rising capabilities and accessibility of open-source models, the paper identifies a critical research gap in addressing threats that exploit full model access. To this end, the authors suggest embedding Space Attacks, operating directly on the continuous representation of input tokens (embedding space), which can bypass conventional safety alignments. The authors conducted extensive experiments to demonstrate the effectiveness of their attacking strategy.

Strengths

1. The authors Introduce a new type of adversarial attack that has been underexplored, providing a fresh perspective on security in open-source LLMs. 2. As open-source models become more prevalent and powerful, the research focuses on the safety of the open-source LLMs is an urgent need for comprehensive security strategies. 3. The study includes experiments across multiple datasets and models, more or less revealing the credibility and generalizability of the results.

Weaknesses

1. My major concern is in whether the suggested technique truly elicit the parameterised knowledge once deleted. All in all, the method can be viewed as a fine-tuning method with only very a few of learnable parameters (i.e., the embedding perturbation). Then, considering the fact that fully-supervision is used to trained such perturbations, one cannot figure out the model learned from some of the new knowledge or elicit the originally deleted knowledge. 2. More detailed discussion about the application scenarios are required. Based on my first concerns, I wonder if the suggested method can truly reflect the sensitivity of the models. If so, is there any practical applications where strong supervision are required. 3. For knowledge that we know it does not exist in the LLM in advance, I wonder if the suggested method will mistakenly recover this knowledge. If so, I do not think it is a proper attacking method. 2. The concreteness of the presentation can be further improved, more discussion about the applications of the suggested attacks, detailed discussion about the metrics, datasets, and unlearning methods are also of my interest. is about the applications of the suggested attacking methods. It seems that the success of the attacks require

Questions

Please refer to the Weaknesses. Kindly please forgive me if I made my misunderstanding. Thanks!

Rating

4

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

Please refer to the Weaknesses.

Summary

This paper proposes a new white-box adversarial attack on large language models (LLMs). The attack is the first to be performed directly in the embedding space of the model; as such, the chosen threat model mainly targets open-source LLMs. The proposed methodology is applied to two goals: (i) removing guardrails for safety aligned models and (ii) showing that unlearned information is not actually forgotten and can be extracted from models.

Strengths

- The paper is well-written, clear and structured. - The evaluation proposed in the paper is extensive. - The proposed attack is effective. - The analysis of unlearned models and its results are novel and shed more light on the functioning of LLMs.

Weaknesses

- The paper has little methodological contribution. The proposed attack is strongly inspired by what the adversarial examples community has been doing for the past decade (here, some iterative version of FGSM without budget constraints). The results presented in this direction are predictable and of limited interest. - The setup itself of attacking open-source models to remove safety alignment also seems limited. Why go through any of this trouble for open-source aligned models when one can just use a non-aligned model to the same end? - The paper should clarify if the attack changes the standard user input by additive noise or concatenation; it looks like concatenation as a suffix is the only scenario covered. - A more extensive comparison with existing input space attacks should be provided (some preliminary results are shown in Tab. 1). - Additional proofreading seems necessary. Minor: - L114 "efficency" -> "efficiency" - L197 "sensibility of the information" -> "sensitivity of the information" - Please adjust y-scale in Fig. 3 to improve readability, as no attack has a low success rate.

Questions

Please answer the points above.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Ok

Summary

This paper introduces embedding space attacks as a novel threat model for open-source large language models (LLMs). The authors demonstrate that these attacks can efficiently circumvent safety alignments and extract supposedly unlearned information from LLMs. The paper presents two main applications: 1) breaking safety guardrails in aligned models, achieving higher success rates and computational efficiency compared to fine-tuning, and 2) extracting information from unlearned models, outperforming standard prompting techniques. The authors conduct experiments on multiple open-source models and datasets, including a custom Harry Potter Q&A benchmark and the TOFU unlearning dataset. They also propose new evaluation metrics such as the cumulative success rate for assessing unlearning quality.

Strengths

- I think the method is a useful tool to have in the red-teaming toolkit. - The multilayer attack, inspired by the logit lens, is interesting. - The approach of creating discrete inputs from continuous embeddings is also curious (and works non-trivially on Llama 2) - The results seem good (but see the concern below about the evaluation), and it’s nice to see that some unlearning methods can be easily broken. - The paper provides further evidence that securing open-weights models against jailbreaking is basically impossible.

Weaknesses

- My main concern is the evaluation methodology. While for measuring the ASR of attacks like GCG, it might be still appropriate to use a keyword-based ASR, I don’t think it’s a good approach for the embedding space attack. In my own experience, embedding space attacks **often produce off-topic results**, so it’s especially important to have an accurate judge like GPT-4 used in the PAIR paper, the HarmBench judge, or the Llama 3 judge from JailbreakBench. And I’m not sure if the toxic-bert model is really good at this - at least, it’s not a standard judge in the jailbreaking literature, and it’s hard to interpret the toxicity results. - In the fully white-box threat model, such as for open-weights models, why not just use prefilling? E.g., the claim *“achieving successful attacks orders of magnitude faster than prior work”* doesn’t hold if one considers the prefilling attack (https://arxiv.org/abs/2404.02151) which is very simple, requires no gradient updates or iterative optimization, and leads to very high attack success rates. I’ll put a borderline reject for now, but I’m ready to increase the score if the evaluation concern is resolved.

Questions

- Llama Guard is the only LLM that is supposed to simply output “safe”/”unsafe” (+ potentially a violation category). Does the embedding attack flips safe to unsafe and vice versa? I don’t think the paper mentions anything specific about Llama Guard.

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The main limitation of the lack of accurate evaluation of ASR is acknowledged, which does seem like a key missing point to me (see above). Another limitation which is not mentioned is the reliance on gradient-based optimization, which limits the scale at which experiments can be performed (i.e., only up to 7B as in this paper).