Summary
This work examines the specific direction within the internal activations of large language models (LLMs) that govern their refusal behavior. Using the difference-in-means technique, the researchers identify this direction and subsequently utilize it to manipulate model behavior in two ways: bypassing refusals for harmful content and reinforcing refusals for harmless content. The identified direction is also employed to update model parameters, resulting in a novel jailbreaking technique that performs comparably to existing methods. Additionally, the research investigates the effect of appending suffix tokens on suppressing refusal behavior.
Strengths
1. The ability to refuse to generate harmful content is crucial for ensuring the safe deployment of LLMs. This work, which enhances our understanding of the internal mechanisms responsible for refusal behavior, is a step in the right direction toward deploying more robust and reliable LMs.
2. The results showing the effectiveness of adding and ablating the identified “refusal” direction is significant and generalizes across 13 models of different sizes and alignment fine-tuning.
3. The work interprets the internal mechanism of refusal behavior and uses the insight gained to propose a novel jailbreaking approach that is on par with other existing techniques. It’s a nice example of the application of interpretability research.
Weaknesses
1. The main contribution of this study is the identification of the "refusal vector." However, the experiments presented do not conclusively demonstrate that this identified vector specifically encodes "refusal" rather than a related concept such as "harmfulness." It is possible that models first determine whether input content is harmful and then use this information in subsequent layers to trigger appropriate refusal responses. As a result, manipulating a "harmfulness" vector could potentially produce similar output behavior as manipulating a "refusal" vector. Therefore, the current experimental results do not provide convincing evidence that the identified vector is indeed a distinct "refusal" vector.
* Intuitively, it is reasonable to hypothesize that the identified vector may be encoding "harmfulness" rather than "refusal." as the contrastive examples used in the discovery process primarily differ in their level of harmful content.
* The hypothesis that the identified vector encodes "harmfulness" rather than "refusal" is further supported by the results presented in section 5.2. The top attention heads, which show the highest feature attribution with the identified direction, primarily focus on instruction tokens containing harmful content. This suggests that these heads are more likely to encode harmful information. When an adversarial suffix is added, the attention of these heads shifts to suffix tokens that do not encode harmful content. Consequently, the harmfulness in their output and in the residual stream decreases. This observation aligns with the findings in section 5.1 (figure 5), which show that the addition of suffix tokens reduces the cosine similarity between the residual stream vector and the identified vector, now presumed to be a "harmfulness" vector rather than a "refusal" vector.
* An experiment to differentiate between “harmfulness” and “refusal” direction is to use Sparse Autoencoder features of the residual stream vector which are used to form the “refusal” direction using the difference-in-means technique. Analyzing the correspondence of these features with harmful and refusal examples could provide insight into the encodings of the identified vector. Furthermore, the authors are encouraged to devise additional experiments to distinguish between the "harmfulness" and "refusal" directions.
2. The analysis of adversarial suffixes presented in Section 5 of the study is limited by its reliance on a single adversarial example and one model, significantly restricting the generalizability of the results (as also mentioned in the text). Additionally, Section G highlights the difficulties encountered in identifying suffixes that are universally effective across different prompts. This challenge indicates that various suffixes might employ distinct subspaces or mechanisms to suppress the model's refusal behavior.
3. The paper's clarity and presentation could be enhanced in several ways. Notably, Table 1 and Figure 2 are not referenced within the main text, which may hinder the reader's ability to connect these visual elements with the relevant discussions. Figure 6 is potentially misleading at first glance. Specifically, the projected value of head H12.8 in Figure 6(a) is ambiguous - it's unclear whether this value is 0.8 or 1.8. Regardless of the intended value, alternative plot types, such as bar plots, could be more effective in conveying this information clearly.
Questions
1. Section 2.1 defines “post-instruction tokens”. Could you give an example and/or more information regarding the actual template being used by the chat models?
2. Why did you decide to use the “difference-in-means” technique to find the subspace rather than using techniques like Distributed Alignment Search (DAS) [1] or Boundless DAS [2], which not only involve causal interventions but also have been more ubiquitous in existing peer-reviewed works?
3. Figure 2 mentions “President James R. Johnson”. However, there is no US president named James R. Johnson, which makes me wonder if ablating the identified “refusal” direction promotes hallucination. This hypothesis is also bolstered by the results in Table 3, suggesting that the orthogonalized model’s performance degrades on TRUTHFULQA. I would recommend updating this example.
4. I wonder about the impact of alignment fine-tuning on the refusal direction. Is it the case that the refusal direction is already present in the base model and fine-tuning is just enhancing it, as suggested by [3]? If so, then probably we don’t need to do expensive alignment fine-tuning. We could make mechanistic updates in the base model parameters to improve its alignment, which would significantly be more efficient than fine-tuning!
[1] Geiger et al, "Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations", 2024.
[2] Wu et al, "Interpretability at Scale: Identifying Causal Mechanisms in Alpaca", 2024.
[3] Prakash et al, "Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking", 2024.
Limitations
The authors have appropriately acknowledged various limitations of their work, encompassing both methodological aspects and empirical findings. However, there are additional limitations that I have outlined in my response to the Weaknesses section, which are not addressed in the paper.