Summary
This paper introduces an activation-intervention method for detoxification of language models. In particular, they (1) sample unconditionally from the target LM and filter for toxic examples, (2) use a much larger instruction-tuned model to produce non-toxic paired examples, (3) estimate differences in layerXattention_head activations between toxic and non-toxic examples, (4) estimate which (layer, attention head) best represents toxicity, (5) add in the non-toxic minus toxic activations average vector during inference to detoxify.
The paper explores various LMs, including GPT-2-large and LLAMA-2-7B. The results are strong in detoxification, and there is a LLM-based preference evaluation showing that their method’s outputs are preferred over other detoxification methods.
There are some interpretability claims in the paper (see below.)
Reasons to accept
Overall, I find this paper to be a nice study in activation arithmetic, model intervention, and generating paired data that shows a simple recipe for a surprisingly powerful intervention to model behavior. The high-level experimental designs – model choices, additional experiments, baselines, etc – all make sense to me.
In reasons to reject, I’ve written quite a bit – there are a lot of lower-level (and some more important) details that make this paper weaker. However, my overall read is that the basic claims of the paper are well backed up by experiment, the method is relatively simple and useful, and the recipe provided might be of interest to a wide range of COLM attendees and general readers.
Reasons to reject
I’m concerned that PPL under GPT-2-XL is used to estimate fluency; this is not a very good model, and e.g., highly repetitive nonsensical text will achieve very low PPL under the model. But the inclusion of LLMs as judges does help alleviate this.
It’s unclear that the probing-decided per-attention-head weights are useful; the ablation in the appendix (as far as I can tell) doesn’t compare to the proposed method. Or if the two tables are comparable, then the fixed-to-0.5-ablation method seems pretty much the same as the proposed probing-based method. So, from the data provided, I don’t really see the head-wise weights as being very important to the method. A lot of space in the paper is given to this head-wise weighting through probing, so this is a bit of a red flag to me, but I’m not giving the paper a lower score because I’d even prefer if the no-probing method worked just as well, since it’s simpler. If the authors want to keep their arguments about disentanglement estimation through probing, though, I really think they need to make it clearer that it actually helps improve the method over a baseline.
Unfortunately, the authors try to use t-SNE to validate that there is a linear representation of toxicity within a representation space. This experiment, in which the post-tSNE space has linear separation, says nothing about linear separation in the original space, because tsne is a highly non-linear dimensionality reduction method. I strongly recommend that this claim be removed, or that the plot be replaced with a linear dimensionality reduction method, e.g., PCA, and linearity checked there.
The methods background 3.1 and methods description in 3.2,3.3 are too vague to get a precise understanding of what activations are being acted upon. (e.g., is it the head-level lower-dimensional vectors? I’d imagine so, but the stated math seems to suggest the full-dimensional token vectors before reshaping to dim//heads.)
The motivation of the linear representation hypothesis seems somewhat confused, especially given section 3.3, wherein the authors state that linearity is at best an approximation due to “false correlations” or “biases” in the dataset, and that “entanglement of toxicity with unrelated attributes varies significantly across different activation spaces” all without citation or data to explain or provide evidence. There are a few instances of this kind of somewhat opaque paragraph in this paper that I find difficult to square with the otherwise straightforward method.
Questions to authors
Please fix many of your citations, which do not have proper spacing.
Saying layer normalization isn’t crucial to your analysis is odd just before providing an equation for the value of a vector under a transformer at a layer and index, since the layer normalization is considered very important in that equation.
> Intuitively, to achieve detoxification without compromising the internal knowledge, the activation space at the output location of the MHSA should be the most optimal.
I don’t share this intuition. Can you explain?
When you say the activation space of each attention head, I’m confused – do you mean the model_dim//head_size dimensional vectors that are computed during MHA, but not passed through the rest of the model?
Is the probe really p(a_i^l) = sigmoid(a_i^l)? This both (1) includes no learnable parameters, and (2) has an output space of the dimensionality of a_i^l, not, e.g., 2 or 1, to allow for classification of toxic/non-toxic. Do you mean p(a_i^l) = sigmoid(w^\top a_i^l) for some parameters w?
In equation 6, why add 1 to \alpha_\prob? As opposed to 0 or some other constant?
> Nonetheless, this methodology, predicated on Probing for weight distribution, facilitates a heightened focus on activation spaces that exhibit a higher degree of disentanglement.
I disagree; just because the activation space is better at predicting toxicity does not mean it’s not also better at linearly predicting various other properties. In fact, many probing studies find that layers that best-encode one property often best-encode other properties as well. (e.g., Liu et al., 2019. Linguistic Knowledge and Transferability of Contextual Representations).
On page one you claim there’s a need for an “interpretable” method for detoxification – however, despite the brief experiments in 5.2, I don’t really see how this method is any more interpretable than, say, finetuning – and the claim isn’t really backed up, so I’d encourage you to remove that from page 1.