Despite careful safety alignment, current large language models (LLMs) remain vulnerable to various attacks. To further unveil the safety risks of LLMs, we introduce a Safety Concept Activation Vector (SCAV) framework, which effectively guides the attacks by accurately interpreting LLMs' safety mechanisms. We then develop an SCAV-guided attack method that can generate both attack prompts and embedding-level attacks with automatically selected perturbation hyperparameters. Both automatic and human evaluations demonstrate that our attack method significantly improves the attack success rate and response quality while requiring less training data. Additionally, we find that our generated attack prompts may be transferable to GPT-4, and the embedding-level attacks may also be transferred to other white-box LLMs whose parameters are known. Our experiments further uncover the safety risks present in current LLMs. For example, in our evaluation of seven open-source LLMs, we observe an average attack success rate of 99.14%, based on the classic keyword-matching criterion. Finally, we provide insights into the safety mechanism of LLMs. The code is available at https://github.com/SproutNan/AI-Safety_SCAV.
Paper
Similar papers
Peer review
Summary
The paper introduces a Safety Concept Activation Vector framework designed to identify and exploit vulnerabilities in LLMs by accurately interpreting their safety mechanisms. The authors develop an SCAV-guided attack method that enhances the success rate of generating harmful prompts and embedding-level attacks with optimized perturbations, requiring less training data.
Strengths
1. The paper is well-written and easy to follow. 2. The evaluation is comprehensive and the results look promising. 3. The discussion on section 4.3 is insightful.
Weaknesses
The idea of inspecting the hidden representation of the model about the safety concept is not novel, which is already proposed in RePE. The difference is that the authors propose to use the SCAV to find the subspaces for benign and harmful prompts, while RePE uses the PCA. The results in Figure 3 look confusing. When only one pair of benign-harmful instruction is presented in the training data, the success rate of the SCAV is already over 80%. It is not clear why it can learn the subspaces so well with only one pair of data. With only one pair of data, the subspace boundary should be very uncertain. The authors should provide more explanation on this. The second concern is that directly modifying the intermediate representation of the model may lead to the model's performance degradation. Modifications to any internal layer will influence all the following layers' representations[1,2]. Without drift control, it is hard to guarantee that the model's performance will not be affected. The authors should provide more discussion on this. Also, the langauge flaw determined by GPT-4 is not a good metric to reflect that. I would suggest using more principled metrics such as weighted average of bi- and tri-gram entropies. [1] Locating and Editing Factual Associations in GPT [2] MASS-EDITING MEMORY IN A TRANSFORMER Other issues: Although the paper claims that they have done comfort for annotators, I think the IRB approval or equivalent is still needed since it involves human subjects and may do potential harm to the annotators.
Questions
The transferability to GPT-4 is questionable, since the embedding editing only works for the local model while GPT-4 has a definitely different embedding. Could you give an reasonable explanation for that?
Rating
4
Confidence
5
Soundness
3
Presentation
3
Contribution
3
Limitations
Yes
Summary
This paper introduces the Safety Concept Activation Vector (SCAV) framework, which guides attacks on LLMs by interpreting their embedding-level safety mechanisms. This work estimates the likelihood that an embedding is considered malicious by the LLM, utilizing a linear classifier based on Concept Activation Vector principles. The attack method generates both attack prompts and embedding-level attacks with automatically selected perturbation hyperparameters. The method improves attack success rates and response quality while requiring less training data. The experimental results show that it's possible to transfer attack prompts across different models, including black-box ones like GPT-4. The findings suggest that existing unlearn methods may not fully erase harmful knowledge, highlighting the persistent safety issues in LLMs.
Strengths
The paper's originality lies in its application of the SCAV framework to guide attacks on LLMs, offering a unique perspective on understanding and exploiting their safety mechanisms. The quality of the work is high, supported by comprehensive experiments that validate the effectiveness of the SCAV framework across multiple models and datasets.
Weaknesses
One significant weakness of the paper is its reliance on the assumption of linear separability, which may not consistently hold true in the complex, high-dimensional spaces characteristic of LLM embeddings. The authors were correct to cite many papers on interpretability that also assume the linear interpretability assumption but to me this is a generally unsound assumption in the interpretability community. In section 2.2 you empirically evaluate linear interpretability with Llama and Vicuna, but while the results show high test accuracy in later layers, the dynamics of how linear separability develops across layers could be explored in more depth. The intuition in 2.3.1 is an oversimplification of high-dimensional spaces. Each layer in a deep neural network applies a non-linear transformation to the data. While later layers might exhibit some linear separability due to the network's hierarchical feature learning, without a mathematical explanation it is not clear to me why non-linear transformations and complex concepts would be accurately captured linearly. The regularization terms in the objective function ( \( \frac{\lambda_1}{2}||w||^2 + \frac{\lambda_2}{2}b^2 \) ) help prevent overfitting, but their choice and tuning are not discussed in detail. Improper regularization can lead to either overfitting or underfitting, affecting the classifier's ability to generalize. While the theoretical foundation for using Concept Activation Vectors is sound, the paper does not provide sufficient empirical evidence to validate this assumption across a diverse range of models and layers. This oversight raises questions about the general applicability and robustness of the SCAV framework. The lack of validation makes it difficult to ascertain whether the linear separability assumption can be reliably applied to different LLM architectures or whether it is specific to the models and datasets used in the experiments. Furthermore, the decision to focus attacks on layers with high test accuracy appears somewhat arbitrary and lacks justification. The authors need to provide a deeper exploration of why these layers are chosen and how this choice impacts the effectiveness of the SCAV-guided attacks. A more detailed discussion on the selection criteria for hyperparameters, such as the thresholds \( P_0 \) and \( P_1 \), would significantly enhance the robustness of the methodology. The current approach leaves the impression that the layer selection and hyperparameter tuning are based on heuristic rather than principled optimization, which undermines the credibility of the proposed framework. The experiments are predominantly focused on a narrow set of models (e.g., LLaMA-2-7B/13B-Chat) and specific datasets (e.g., Advenbench and StrongREJECT). This limited scope constrains the generalizability of the findings. For example, the authors utilize GPT-4 for computing 209 ASR-answer, ASR-usefulness, and Language flaws in the experiments, but not for tables 1 and 2. They could also use Harmbench (Mazeika, Mantas, et al.) as another dataset. To convincingly demonstrate the efficacy and versatility of the SCAV framework, the authors should expand their evaluation to include a broader range of LLM architectures, including those with different training paradigms.
Questions
1. Could you provide more empirical evidence to support the linear separability assumption across different models and layers? This would help validate the applicability of the SCAV framework. 2. How do you determine the thresholds P0 and P1 for attacking specific layers? 3. Have you considered testing the SCAV framework on models with adversarial training or other defensive mechanisms?
Rating
4
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
N/A
Summary
This paper proposes a jailbreak attack (SCAV) inspired by the Concept Activation Vector (CAV) on neural networks. For safety concept, this concept vector is essentially defined as a direction orthogonal to the decision boundary of a linear classifier trained to distinguish safe and harmful instruction on embeddings at a given layer. SCAV works by perturbing the embedding in the direction of this concept vector (towards malicious and away from safe instructions). Under the assumption that the concept “score” is linear, perturbing in concept vector direction is optimal. However, for token-level SCAV, the same technique does not apply so the authors come up with an optimization objective inspired by the embedding-level approach and use AutoDAN’s genetic algorithm to optimize it. Empirical results are convincing at both levels and show some good transferability to GPT-4.
Strengths
### Originality There are several works like JRE and RepE that attempt to mount a jailbreak attack at the embedding levels using an interpretability technique. However, this work fixes an obvious problem of these methods with a simple technique. It is also well-motivated by a classical interpretability method (albeit with a very strong assumption). ### Quality All the formulations in Section 2 are technically sound given the linearity assumption. Algorithm 1 also makes a lot of sense. The experiments are relatively thorough (see the Weaknesses section for some suggestions) with a good number of baselines, different target models, transferability results, and some ablation studies. Among the other sections, I find Section 2.3 particularly clear and convincing. ### Significance Strong and efficient jailbreak attacks help safety evaluation. This is a significant and timely research problem in my opinion. I also like the fact that this attack algorithm seems to also help us learn about the concept activations in LLMs (such as their linearity). The technique itself can also apply to other use cases beyond safety (concept erasure or unlearning is one other use case explored briefly in this paper). ### Clarity The paper is well-written and easy to follow in all of the sections. I like the paper structure as well as how all of the ideas and results are presented.
Weaknesses
### 1. Choice of layer to compute SCAV objective I would like to see an ablation study on the choice of layer being optimized. Perhaps, Figure 4(a) does cover a bit of this, but I have a few more questions. 1. Why is only the last layer optimized in the token-level attack (Eq. (5)), unlike the embedding-level where the objective covers the prior layers? From Eq. (5), it is possible to optimize over $e_S^l$ while the objective is computed on any other layer $e_S^{k}$ where $k > l$. There seem to be some design choices here that were not explored. Presumably, if the objective depends on the last layer anyway, including earlier layers in the objective should not increase the overall computation. 2. What would happen if only the last layer is perturbed in the embedding-level attack? Is there a benefit from optimizing many layer from an early one to the last? 3. For both attacks, I would like to see how ASR changes with the choice of layer, given that only one layer is perturbed. ### 2. Token-level SCAV objective I’m curious why the authors decide to use the objective in Eq. (5) instead of the Lagrangian relaxation of the optimization problem in Eq. (3) into something like the following: $$ \arg\min_{e^L_S}~ \lvert| e^L_S - e^L \rvert| + \lambda P_m(e^L_S) $$ Is there an advantage to the product form instead of a weighted sum? ### 3. Missing baseline attacks 1. **Comparison to soft prompt attack for embedding-level SCAV.** While the authors already compare SCAV to JRE and RepE, I believe it is also important to compare to other soft prompt optimization attack (such as https://arxiv.org/abs/2402.09063) as the concept is very similar. These soft prompts (can be prefix or suffix) are also known to be very successful against open-source LLMs. 2. **Comparison to GCG in Table 3 & 4 for token-level SCAV.** Unless there is a very good reason, I believe that it is important to have GCG as a baseline for any token-level attack, even if AutoDAN claims to be better than GCG in some scenarios. GCG is still the most popular attack at this point so it allows for easy comparison / verification across different works.
Questions
**Q1:** For the embedding-level SCAV, when an embedding $e^l$ is mentioned (e.g., L115), does this refer to (1) embedding of one of the prompt tokens or (2) a newly added prefix (like soft prompt tuning) or suffix (like GCG but soft) token? I just want to make sure I understand the detail here, but it doesn’t seem clearly defined in Section 2.1. **Q2:** This might be a follow-up question to Q1. I’m not very convinced by the statement on L121-122: > The first term that minimizes $|\epsilon|$ ensures a small performance loss of LLMs, avoiding flaws such as repetitive or irrelevant responses. > unless the optimized embedding $e^l$ is from one of the prompt tokens. So I assume that $e^l$ is the next-token embedding (i.e., from the last input token)? **Q3:** Can the authors share details on how Figure 2 is created? Or is it more like a diagram? **Q4:** What’s the difference between ASR-answer and ASR-useful? **Q5:** In Table 7 (Section 4.2), Is the SCAV attack here at embedding or prompt level? Since the baselines are GCG and AutoDAN, I’m assuming that this is at a prompt level?
Rating
5
Confidence
4
Soundness
3
Presentation
4
Contribution
3
Limitations
Limitations and negative societal impact have been adequately addressed.
Summary
This paper introduces a new framework named Safety Concept Activation Vector (SCAV) for jailbreaking LLMs. This framework is built upon LLM interpretability work. Specifically, the paper utilizes an interoperability approach called Concept Activation Vector, which can linearly separate safe and unsafe instructions in the latent representation space. Given this linear separability, this paper proposes that a jailbreak attack can be achieved if one can make perturbations such that the latent feature of an unsafe prompt is perturbed to the feature subspace of a safe prompt. Following this idea, the authors show the feasibility of doing such jailbreak attacks in both the feature space and the prompt space.
Strengths
1. The paper is well-written and well-motivated. The presentation is also clean and comprehensive. 2. The success rate of the attack is good, showing good improvement over existing baselines. The improved transferability to attack GPT-4 is also an advantage of the attack. 3. The paper also shows that existing unlearn methods fail to defend against the proposed attack, suggesting that unlearning may not be an effective solution to stronger jailbreaking attacks.
Weaknesses
1. The idea of using an interpretability approach to assist jailbreak attacks is not new and has already been explored by [1]. 2. It would be better if the authors could also discuss why the proposed attack can be more effective than previous ones and what are possible ways to mitigate the attacks. [1] Zou, Andy, et al. "Representation engineering: A top-down approach to ai transparency." arXiv preprint arXiv:2310.01405 (2023).
Questions
Can the authors clarify more on how the prompt space attack is implemented? In the paper, the authors only mentioned the use of AutoDAN’s hierarchical genetic algorithm to solve the optimization. It would be better if more details could be clarified. For the representation space attack, the adversaries can directly interfere with the internal components of the model. Would this be closer to the threat model of fine-tuning attacks presented in [1]? In that case, it seems that fine-tuning attacks are easier and more effective. It would be good if the authors could discuss the connection. [1] Qi, Xiangyu, et al. "Fine-tuning aligned language models compromises safety, even when users do not intend to!." arXiv preprint arXiv:2310.03693 (2023).
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
No major limitations were found.
Thanks Authors
I would like to thank the authors for their detailed responses. I believe this paper makes a good improvement of LLM jailbreak attacks, which could be used to further encourage future development of stronger defenses. Thus, I will keep my initial recommendation of weak accept. Meanwhile, I noticed the authors repeatedly use the term "misleading heuristics" throughout the paper and the rebuttal context. This term is very hand-wavy. I suggest the authors be more precise on what this means, which may make the audience easier to understand. I also noticed that the authors reported higher ASR than fine-tuning attacks. Can the authors share their intuition on why an embedding-level attack would be even stronger than fine-tuning that directly modifies model weights?
Thanks for your comment!
Thanks for your comment! **Clarifying the specific meaning of the term “misleading heuristic”** Thanks for pointing out the potential confusion caused by using this term. We are glad to provide a clearer and more factual explanation of what this term means. - We observed the linear separability in embeddings (Figure 1) and the relationship between jailbreak ASR and $P_m$ (Table 9), which guided the perturbation vector learning. In comparison, the extraction methods used by RepE and JRE are assumed heuristically. - We found that RepE and JRE sometimes cannot correctly extract the vector direction as described by their heuristic (Figure 2). Moreover, in a small number of training sample scenarios, these two baselines result in a lower ASR (Figure 3) because of this. Following your suggestion, we will provide more precise explanations in our paper, instead of just using this term to summarize. **Intuition on why SCAV-embedding outperforms fine-tuning attack** We are glad to share our intuition on why SCAV-embedding performs better than fine-tuning attacks with you. - Fine-tuning directly modifies the weight of models, which cannot adaptively modify their behavior towards specific malicious instructions. However, our paper's Algorithm 1 can ensure accurate and effective attacks by adaptively modifying the embeddings of each layer. - LLMs obtain safety concepts through a large number of corpus fine-tuning after pre-training. A small amount of corpus used for fine-tuning attacks may not be generalized enough to bypass all the capabilities of the safety concepts. However, SCAV-embedding attacks directly modify the safety concepts in embeddings to reverse LLMs’ recognition of malicious instructions to safe ones, thus leading to higher ASR. Thank you for praising our work in the comments. We hope the above content can clarify your doubts. We are glad to continue to provide you with any details.
IRB approval obtained
Thank you for your valuable feedback on our submission. We have obtained IRB approval from ethical review office of our annotation service vendor. We are confident that our project meets the necessary ethical standards, and we are glad to provide approval email for your reference. --- ### **Ethical Approval Confirmation for Your Human Annotation Process** Dear *[Masked due to the anonymous policy]*, I am writing to confirm that our ethical review office has reviewed and approved the human annotation project associated with your recent research work on unveiling safety risks of large language models. The review process was conducted in accordance with our stringent informed consent management requirements. We understand that this project involves human annotators assessing content generated by LLMs to identify potential safety risks. The office carefully evaluated the design of the annotation process, noting the informed consent process were thoroughly designed and implemented. The office also appreciated the follow-up procedures in place to ensure that the annotation process did not negatively impact participants. Based on this comprehensive review, our review office is satisfied that the project meets our informed consent management standards, with appropriate safeguards in place for all human participants. Please let us know if you need further documentation or additional support. Thank you for your continued collaboration. Best regards, *[Masked due to the anonymous policy]* Director *[Masked due to the anonymous policy]*
Dear Reviewer CRgs, Thank you for your thoughtful feedback on our submission. We kindly remind you to review our rebuttal and let us know if it adequately addresses your concerns. If you believe our explanations and revisions have satisfactorily resolved the issues, we would greatly appreciate it if you could reconsider your evaluation of our paper. Thank you again for your time and for providing valuable guidance. If there are any further questions or suggestions, we are glad to discuss them with you. Best regards, Authors of paper 7016
Dear Reviewer mJhn, Thank you for your thoughtful feedback on our submission. We kindly remind you to review our rebuttal and let us know if it adequately addresses your concerns. If you believe our explanations and revisions have satisfactorily resolved the issues, we would greatly appreciate it if you could reconsider your evaluation of our paper. Thank you again for your time and for providing valuable guidance. If there are any further questions or suggestions, we are glad to discuss them with you. Best regards, Authors of paper 7016
Dear Reviewer xmcv, Thank you for your thoughtful feedback on our submission. We kindly remind you to review our rebuttal and let us know if it adequately addresses your concerns. If you believe our explanations and revisions have satisfactorily resolved the issues, we would greatly appreciate it if you could reconsider your evaluation of our paper. Thank you again for your time and for providing valuable guidance. If there are any further questions or suggestions, we are glad to discuss them with you. Best regards, Authors of paper 7016
Dear Authors, Thank you for your rebuttal on your paper. While I appreciate your efforts to clarify the contributions and methodologies, I still have some concerns that prevent me from raising my score. Below are the reasons: **Novelty and Superiority of SCAV**: The use of interpretability in understanding the hidden representation of safety concepts is not novel, as similar approaches have been explored in previous works like RePE. Your paper claims that SCAV is superior to PCA or other clustering methods. However, there is no statistical analysis provided that demonstrates SCAV's superiority over these methods. Without such evidence, it is difficult to convincingly argue that SCAV offers a significant improvement or new insights over existing techniques, or it is a general discovery that could generalize to other fileds related with hidden representation inspection. Since the separation of harmful and harmless prompt representations is already a known phenomenon, this aspect of your work does not bring new insights to the community. **One-Pair Classification**: The claim that a single pair of training data can achieve high classification accuracy raises further questions about the novelty and contribution of your approach. If representations are already well-separated, as demonstrated by your one-pair classification, it suggests that any clustering method could achieve similar results. This undermines the argument that SCAV is necessary for superior performance and conflicts with the claim of SCAV's distinct advantage. **Transferability Without Explanation**: While it is recognized that other works have attempted to transfer white-box attacks to black-box models, these claims often lack a sound explanation for why such transfers work. For instance, while GCG reports good transfer performance, subsequent works like AutoDAN show very low ASR for transfer GCG. Your paper should compare SCAV with existing black-box methods for jailbreaking and provide a deeper explanation of why the transferability of your method to models like GPT-4 is expected or reliable. Without this, the work does not provide new insights into the understanding of transferability for the community. Given these concerns, I believe that your work does not sufficiently address these issues to warrant an increase in my evaluation score.
Clarify factual misunderstandings in your comment [1/2]
Thanks for your detailed feedback. We appreciate your efforts in clarifying your concerns. We would like to take this opportunity to clarify some points that may have been misunderstood. ### **Novelty and Superiority of SCAV** You said that "there is no statistical analysis provided that demonstrates SCAV's superiority over these methods." We have demonstrated SCAV’s superiority over existing methods by conducting multiple experiments, and some of experiments are conducted based on your suggestion. Here is a summarization of statical analysis that we provide to demonstrate our superiority: - **Analysis on averaged $P_m$ decrease**: In Table 9, we compute the average accuracy of perturbation direction of existing methods, according to whether RepE successfully lowers the malicious probability $P_m$ according to the linear classifier and the percentage is only 7%-10% compared with 0.01% of our method. These can also be mutually supported by the ASR of each baseline. - **Analysis on ASRs and Language improvement**: In Tables 1-5, we have shown that our method improves ASR over RepE by 12%-30% and JRE by 16%-34%, decreasing the percentage of language flaws by 14%-42%. - **Analysis on performance less data**: In Figure 3, We have shown that in the case of less training samples (1-10), SCAV reached nearly 80% of ASR at 1 sample, while ASR of RepE is 0, and JRE is only 50%. ASR of SCAV reached 100% at 5 samples. And it is consistently significantly better than the performance of RepE and JRE. - **Analysis on vector extraction difference**: In Figure 2, we analyze why RepE and JRE may cause incorrect direction extraction. These data are synthesized according to the real data dimension reduction distribution in Figures 6 and 7, and the arrows are obtained by the real vector extraction algorithms. - **Analysis on t-SNE embedding distribution**: In Figures 6 and 7, we have shown you the distribution of malicious instructions in Advbench and the generated safety instructions embedding after t-SNE dimension reduction. The results show that linear separability is different in different layers, and the middle and late layers are more significant. - **Analysis on which layers to perturb**: In Figures 1, 4(a), 4(b), we analyze the significance of each layer for perturbation. The linear separability of each layer does not always exist or exist independently, and it needs to be determined automatically. These analyses illustrate the advantages of SCAV, especially Algorithm 1. - **Analysis on weighted entropy per your request**: In Rebuttal Table 3, we have shown the language flaws analysis on more principled metrics (weighted average of bi- and tri-gram entropies) per your request. The results still hold our conclusion. If there are further analysis that you suggested, we appreciate your specific guidance and we are glad to provide the results. ### **One-Pair Classification** You said that "any clustering method could achieve similar results", could you clarify which clustering methods you refer to? We have compared with baselines RepE and JRE (Tables 1/2 and Figure 3), and they cannot achieve similar performance even when embeddings are well separated, because: - Well-separation only happens for certain layers and only our method identifies these layers to perform the attack. - **Well-separation does not automatically lead to optimal attack.** Instead, we still need to careful in finding the optimal direction that minimizes the perturbation magnitude, which requires us to understand the separation hyperplane, and this is only performed by using our method. Existing methods that heuristically select a direction cannot always guarantee this.
Clarify factual misunderstandings in your comment [2/2]
### **Transferability Without Explanation** You said that "Your paper should compare SCAV with existing black-box methods for jailbreaking". We have conducted such experiments with black-box methods DeepInception [1] and AutoDAN [2]. They are presented in Tables 3/4 and GCG [3] presented in Rebuttal Tables 4/5 per the request of reviewer CRgs. You said that "the work does not provide new insights into the understanding of transferability for the community". New insights in this aspect is not the focus of our paper. Our goal here is to follow existing methods and show the potential usefulness of method when applied to GPT-4. In terms of insights, we have provided multiple others and reviewers 6xey, CRgs, mJhn have acknowledged as insightful: - "I believe this paper makes a good improvement of LLM jailbreak attacks, which could be used to further encourage future development of stronger defenses" by reviewer 6xey - "this work fixes an obvious problem of these methods with a simple technique" by reviewer CRgs - "offering a unique perspective on understanding and exploiting their safety mechanisms" by reviewer mJhn All insights we have: - The first to observe and report linear separable phenomenon of malicious and safe embeddings, and verify it on 8 open-source LLMs popular in community - Uncovering safety risks by SCAV of open-source models, black-box models and adversarial training models (unlearn [4] and per the request of reviewer mJhn, RR [5]) - The proposed algorithm shows the distribution of safety concept at each layer and reflects the relationship between them [1] "we mainly consider the training-free and **black-box** jailbreak, which is more practical." DeepInception: Hypnotize Large Language Model to Be Jailbreaker [2] "these interpretable prompts generalize better to unseen harmful behaviors and transfer better to **black-box** LLMs than the unreadable ones in prior work." AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models [3] "the adversarial prompts generated by our approach are highly transferable, including to **black-box**, publicly released, production LLMs." Universal and Transferable Adversarial Attacks on Aligned Language Models [4] Eraser: Jailbreaking Defense in Large Language Models via Unlearning Harmful Knowledge [5] Improving Alignment and Robustness with Short Circuiting
Decision
Accept (poster)