Summary
This paper investigates the mechanics of response refusal in LLMs by probing the Llama 2 13B and Vicuna 13B models. The authors investigated two ways of manipulating the model’s response behavior - prompt prefix (PP), or prepending text with instructions to the prompt, and contrastive activation addition (CAA). They present three main findings. First, the paper finds that even when a model refuses to answer an offensive prompt, the offensive information can be decoded from earlier layers, suggesting that the response formation happens in early layers and the refusal happens in later layers of safety-tuned models. Second, the authors show mild success in inducing refusal or response with the CAA method, while the PP method is not able to circumvent the safety tuning. Finally, the authors construct user personas and show that certain types of user are more likely to be able to avoid response refusal, especially using the CAA method. Overall, the paper is a detailed investigation into the dynamics of response refusal in an LLM.
Strengths
This paper’s primary strength is in the thoroughness and depth of its analysis. The authors do a fantastic job of diving deep into these effects, not just measuring their presence but also attempting to disentangle the “why.” The use of steering vectors is a particularly strong contribution to the literature, and the application of Patchscope is also a clever way to begin to explain some of the model’s behaviors. The paper is very well written and presented and quite rigorous, which is wonderful to see.
Weaknesses
This paper does not have many weaknesses, but one area that could be improved is in the motivation or introduction to the problem space generally. More details on what constitutes a problematic prompt, what definitions of safety are relevant in this setting, and generally just more context on the problem would be quite helpful. I understand this was likely a constraint of space given the extensive results that are shown, but for the camera ready version I would suggest adding more intro information.
Another small nitpick is with the language that is used in parts of the paper. I particularly take issue with the phrasing that the model “perceives” user personas. Models don’t “perceive” in the way we typically think of the meaning of the word. I think it might be more accurate to say that the user attributes are fed to the model rather than perceived by them.
Questions
Can you define what “misaligned” means in this context? Similarly, what does “safety” mean for the LLMs in this paper?
Are there any additional tests you could do to ensure the generalization of the observed effects? Is this something you think is specific to Llama and Vicuna or does it go beyond those models?
In practice it seems like one would need to have direct access to the model to circumvent the safety measures using the techniques in this paper. Is that a fair statement? Can you discuss the implications of these findings on potential adversarial attacks on safety-tuned LLMs?
Limitations
Some limitations are discussed in the appendix, but I would suggest moving these to the main body for the camera ready version.