Spectral Editing of Activations for Large Language Model Alignment

Summary

This paper focuses on the problem of editing undesirable behaviours at inference time that does not require any training. To that end, they present SEA, a method based on spectral editing of activations. To find the editing projections, the method requires keeping track of LLM activations over several neutral, positive and negative activations. From those activations, SVD is applied on the covariance matrices between the neutral and negative, and neutral and positive activations, respectively. To allow for non-linearity, the authors use an invertible non-linear feature function. The authors investigate how their method impacts truthfulness and bias on two benchmarks (TruthfulQA and BBQ), and show that scores on those benchmarks can be improved using SEA. Consistent improvements are observed across six distinct LLMs of various sizes and architectures, using only 25 demonstrations, while not degrading other model capabilities, with an increase in inference speed of around 4%. The paper first presents results with Llama-2 7B (base and chat) for thruthfulQA, comparing with several baselines, including ICL and LoRA. Overall SEA outperforms other methods while maintaining a much better inference speed. With an ablation study, the authors show that activations edited with positive and negative projections likely complement each other, and are not as effective on their own. They furthermore investigate the impact of feature normalization, showing that it is more effective than when no normalization has taken place. Next, the paper investigates the impact of SEA on bias, as measured by BBQ. They show that the accuracy enhancement for linear SEA is moderate, but non-linear SEA gives more improvements, while baselines do not. For bia, they furthermore show that the results can be generalised to other Llama models (llama-2-13B & 70B, Gemma-it-2B & 7B and Mistral 7B). Last, the paper investigates how SEA scales with the number of demonstrations needed to calculate the editing projections. Experiments show that for MC1 a mere 25 demonstrations suffice for the first improvements (no results are shown for MC2), for BBQ even fewer demonstrations can improve accuracy (number listed is also 25?), and shows that the method has little effect on a several other benchmarks unrelated to the edited parts.

Strengths

The paper discusses an important topic of making models more truthful and less biased. The proposed methods seems to work better than previous methods (though see weaknesses below) at a smaller loss of inference speed, while maintaining performance on benchmarks unrelated to the editing.

Weaknesses

- There is no significance testing for benchmark scores. Especially TruthfulQA is not a very large benchmark, for the entire benchmark (averaging over subsets), the 95% confidence intervals would be around 3, making several (but not all) of the reported differences insignificant. This should be addressed / discussed - The paper would be stronger if more evaluation benchmarks were considered for bias and truthfulness - Some of the results selection seems a bit arbitrary, which gives pause when considering the generalisability of the results. For instance, why are results for other model (families) shown for BBQ, but not for TruthfulQA? And why are scalability results for truthfulQA shown only for MC1, and not for MC2? - The toxigen scores for llama2-chat-7B seem outrageously high, in the Llama2 paper they are listed to be between 20 and 30 for the pretrained model, and around 0 for the chat version. In table 4, however, the scores are reported to be higher than 50 (!). - It is not entirely clear if the method would scale to making models more truthful and unbiased at the same time, would that require different editing projections to be stacked on top of each other? Some presentational issues: - Figure 4 is a bit difficult to read because of the scale. The text makes statements about values around 25, but this cannot be confirmed from the figure. Perhaps a log-scale ould be more suitable? Or, alternatively, let the plot go up to 50, rather than 1500/2000, as nothing is discussed about values higher than 25 anyways. - In Table 4, toxigen scores going down are reported red, but for toxigen lower is better

Questions

- Could you explain why your toxigen scores differ so drastically from the scores reported in the Llama2 paper? - It could be that I am mistaken, but it seems that several separate editing functions are needed for truthfulness and bias. Can this approach scale to a method where both are taken care of?

Rating

6

Confidence

1

Soundness

2

Presentation

3

Contribution

2

Limitations

The discussion of limitations is very limited, only discussing a specific performance degradation of not-linear SEA on control tasks (which is a limitation indeed, but really more just an experimental result). The limitations section is furthermore not referred to in the main text, but is a far-down appendix.

Summary

This paper introduces Spectral Editing of Activations (SEA), which adjusts the internal activations of LLMs to enhance alignment with truthful and unbiased content. This technique involves projecting input representations to maximize correlation with positive examples (truthful content) while minimizing correlation with negative examples (biased or false content). The method can be applied during inference and is further extended to non-linear editing using feature functions. Comprehensive experiments were conducted on benchmarks related to truthfulness and bias.

Strengths

- Research on representation engineering is very interesting and has great potential. - The experimental part is comprehensive and the effectiveness of the proposed method is evaluated on various benchmarks. - Paper is well written and easy to follow.

Weaknesses

- Some recent works in representation engineering should be included in the article, such as TrFr[1], TruthX[2]. \ In particular, as far as I know, TruthX uses auto-encoder and contrastive learning to learn the editing direction on LLM's representation. This sounds similar to the motivation of "SEA edits activations by keeping them highly correlated with activations associated with positive behavior (e.g., truthful) and decorrelated with negative behavior (e.g., hallucinated)". \ I suggest that the author can compare SEA with these methods in the article to highlight the novelty of the proposed method. \ [1] TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Space \ [2] Truth forest: Toward multi-scale truthfulness in large language models through intervention without tuning \ - Some baselines on representation engineering, such as TrFr and TruthX, should be compared in the TruthfulQA experiment. As far as I know, they all report MC1 and MC2 in their papers (some work open-sources the trained models, because it should not be complicated to evaluate their methods and compare them with SEA). - It is not sufficient to use only multiple choice tasks for TruthfulQA. The authors should further test it on open-ended generation tasks (like previous works), because in real applications we interact with LLM in a conversational manner rather than making multiple choices. - The statement about "training-free" needs to be more rigorous. In my understanding, SEA does not require training LLM, but the process of "Finding the Editing Projections" is actually a training process, but its cost is very small. Like previous work, I prefer to call it "inference-time". - The motivations for some settings lack in-depth explanations and experimental ablations, refer to the Question section. I can understand that the author has some heuristic designs/choices, which is acceptable, so this is not a core weakness. But I suggest that some specific explanations and experimental results (if possible) can be added, which can make this research more insightful.

Questions

- When extracting activations within LLM, why do we use "activations at the last token position" instead of randomly selecting or taking the mean of all activations in the response? - Why use the output of each MLP layer as activation? Instead of using attention head like ITI, or using attention and MLP like TruthX? Looking forward to the author's response, I might consider raising the score if the relevant issues are addressed.

Rating

4

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

The paper introduces a novel method called Spectral Editing of Activations (SEA) to improve the alignment of LLMs by enhancing truthfulness and reducing bias. SEA operates at inference time, projecting input representations in ways that maximize correlation with positive demonstrations (truthful content) and minimize correlation with negative demonstrations (hallucinated content). The method leverages singular value decomposition for linear editing and extends to non-linear editing using feature functions. Extensive experiments on benchmarks for truthfulness and bias demonstrate SEA's effectiveness, generalizability, and efficiency across six different LLMs. The results highlight SEA's ability to improve model performance on tasks like TruthfulQA and the BBQ dataset with minimal impact on other model capabilities.

Strengths

The paper presents a unique inference-time editing method, SEA, which uses spectral decomposition to improve LLM alignment. This approach is novel compared to existing optimization-heavy methods. The experimental design is robust, involving multiple benchmarks and diverse LLMs, demonstrating SEA's effectiveness in improving truthfulness and fairness while maintaining computational efficiency. The paper is well-written and clearly explains the methodology, including the theoretical foundations of SEA and its practical implementation. The use of figures, like the one illustrating activation clusters, aids in understanding the concepts. The ability to edit LLM activations to enhance desirable properties like truthfulness and reduce undesirable behaviors like bias has significant implications for the deployment of more reliable and fair NLP applications.

Weaknesses

The paper could benefit from experiments on a broader array of tasks to further validate SEA's effectiveness across different contexts. This would help in generalizing the findings beyond the current benchmarks. Including visualizations of the distribution shifts in activations before and after applying SEA would provide more insight into the impact of the method and help in understanding the underlying mechanics.

Questions

Can you provide visualizations of the activation distribution shifts before and after applying SEA? This would help in understanding the impact of the method on the internal representations. How does SEA perform on other important NLP tasks not covered in this study? Extending the evaluation to a wider range of tasks could further establish its generalizability. Could you elaborate on the choice of benchmarks and how representative they are of real-world scenarios where LLM alignment is critical?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

While the paper demonstrates SEA's effectiveness on truthfulness and fairness benchmarks, a more comprehensive evaluation across a wider array of tasks and datasets would provide stronger evidence of its generalizability.

Summary

### Summary - This paper presents an inference time alignment algorithm based on activation editing. - Their technique named as spectral editing of activations (SEA) projects the input representations onto directions with maximal covariance with positive demonstrations (truthful) and minimum covariance with negative demonstrations (hallucinations). - They use SVD to to find projection directions which correlate maximally with positive and negative demonstrations. - Equation (1), (2) in the paper describe the technique well. - The idea is to keep the largest singular values for positive demonstrations and smallest singular values for negative demonstrations. - The positive and negative activation vectors after editing are merged together with a feature normalization factor which they show later in the paper is important through ablation studies. - In addition to linear transformations, the authors extend the transformations to non-linear setting. This is based on the hypothesis that certain behaviors like producing biased responses may not exhibit linear separability in the activation space. To this end, they experiment with three non-linear kernels ### nits and typos - Line 239: optimisation -> optimization -

Strengths

## Strengths - Compared to traditional activation engineering methods which require iterative optimization their proposed technique is training free. - The paper is well written and easy to follow. - The experiment evaluating truthfulness and speed is convincing. While I understand that you are considering inference time methods as baselines, I'm curious to know how these methods compare to tuning-based methods like DPO. Do they come close in performance? - The experiment on Bias Evaluation with non-linear function somewhat supports the hypothesis about non-linear separability of bias.

Weaknesses

## Weakness - Line 294 is a strong claim. As shown in Figure 4 BBQ, the performance plateaus. Consider re-phrasing. - Performance on control tasks is generally convincing but need to be careful about applying this technique to common sense tasks. The explanation given in line 307-308 does not tell me why the lossy function, does not apply to math tasks but only selectively to common sense QA. - I am also surprised why the authors did not compare with Best-of-N alignment as a baseline which also does not require any training and is quite simple to compare against.

Questions

## Questions - What is a good working value for the hyperparam K (line 127) ? - How do you enforce that k = r/2 where r is the rank of the matrix? Or is there no such constraint? Otherwise you end up double summing the values for overlapping directions. - It seems from table 1, that you are keeping top 99% and bottom 99% of explained variance. Does this not lead to double summation for activations? - Also this makes me wonder if the spectrum of activations really decays exponentially?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have adequately addressed the limitations.