Interpreting Learned Feedback Patterns in Large Language Models

Summary

This submission tries to tackle one big question in the field of interpreting the data-driven preference learnt by RLHF in human language. The technical path this submission took is to train probe on SAE features to distinguish between good and bad RLHF features.

Strengths

+ The attempt to interpret what happens during RLHF training is a good direction to pursue. + Releasing the SAE direction and training code could be an excellent news to the community.

Weaknesses

+ Unclear why have to probe on top of SAE feature. SAE greatly increase the dimensions of the features, leading to overfitting---you can find a separating plane for whatever classification task in this high-D space. Lacking comparison to normal probing. + Considering the problem from a dynamical perspective can be fruitful. Noted that the authors did ablate the features and observe a performance drop on preference dataset. But it's also interesting to see the progress of RLHF training, how it warps the features spaces, even the SAE features' relative importances.

Questions

+ I wonder if it could be interesting to conduct the same analysis on the reward model, if the reward model is another language model that is open-weight and trained. Can we compare the two representation space, one trained for discrimination and the other trained for generation?

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

2

Limitations

N/A

Summary

The goal of this paper is to predict where patterns in LLM activations learned from RLHF diverge from the human preferences used for the RLHF training. Given a base model and an RLHF tuned version of it, the method involves first identifying the 5 layers with highest parameter difference according to an L2 norm. Then two auto-encoders are trained over the activations from these layers. The encoder and decoder weights of the autoencoder are tied, and the output from these is preferred for studying the activations as they are expected to be more sparse, condensed and interpretable than the raw activations. At inference time, for each input, the activations from the high divergence layers are computed, passed through the autoencoder and then aggregated. Given a pair of contrasting inputs, a linear probe is trained to predict activation deltas using the above aggregated autoencoder output as input. The output of the probe is meant to be a predicted feedback signal that can be compared to the ground truth fine tuning feedback. For sentiment analysis, a strong correlation is observed with the Pythia-160m model but this is weaker for Pythia-70m and GPT-Neo-125m. For another validation the probes, GPT-4 is used to generate explanations of the features in the decoder weights of the autoencoders that get activated when the predicted feedback is positive. GPT4 is then prompted to predict whether or not these are relevant to the fine tuning task, based on a language description of the task. It is found that a feature identified by GPT-4 as relevant to the fine-tuning task is between twice and thrice as likely to be correlated with predicted positive feedback.

Strengths

The paper is quite accessible for a reader whose area of focus is not interpretability.

Weaknesses

As a reviewer not particularly experienced with work on interpretability, the takeaways of this paper are somewhat unclear. For example, if we finetuned a new model on one of the datasets used in this paper and trained probes in a similar way from its activations, what would that tell us about the the difference between the base and RLHF versions of that model? Alternately, is the goal to discover information about a model where the base and RLHF-tuned versions are available but the data is not, and hence we do not know what factors might have influenced the preference annotations that guided the annotation. I did not fully understand how the activation deltas are calculated. While most of the paper is fairly readable to a reviewer with a different area of focus, this aspect could be improved.

Questions

1. I don't feel like I understood the concept of the activation delta. The paper states "We compute the activation deltas for a given contrastive triple as the difference between the positive 212 and neutral element and negative and neutral element under the ℓ2 norm". For any input x, there is a set of values $\hat{a}$. For calculating an L2 norm these still need to somehow be aggregated. Since the probe input $A_{concat}(x)$ is already a concatenation of these values, I assume it is not simply an L2 norm of the two $A_{concat}$ vectors, as then it is unclear what the probe would learn. 2. Assuming that the probes trained in this paper do obtain information about the preferences underlying human preference data, how do we make use of that information?

Rating

5

Confidence

1

Soundness

2

Presentation

2

Contribution

2

Limitations

The paper has discussed limitations but not broader impact of their method.

Summary

The authors propose an approach for measuring and interpreting the divergence between learned feedback patterns (LFPs, or simply the model's activation patterns) and the feedback reward distribution of the preference training dataset. To do so, they identify layers whose activations have moved the most during RLHF training and input these layers' activations into a sparse auto-encoder (SAE) that is trained to provide sparse representations of the LLM's activations. Then, they train probes to predict the feedback signal (e.g. reward, sentiment label) from the SAE's outputs. They use these probes both to measure the divergence of the LFPs from the actual feedback signals and to interpret which features are most important for the LFPs.

Strengths

- The authors ask an interesting question of whether we can measure and interpret the difference between a trained model's activation patterns and the preference distribution it has been trained on. The interpretability aspect of this question is interesting, since it can help us better understand what exactly the model has learned (or not learned) from its training dataset. - The authors provide a good explanation of why sparse auto-encoders are being used for this task (rather than interpreting the raw model activations), as well as the limitations thereof.

Weaknesses

- The effectiveness of this probing method seems to rely on many key assumptions being true, such as (i) sparse autoencoder outputs being more interpretable than the original model's outputs, (ii) sparse autoencoder output representations being faithful to the original model's representations, (iii) the probes being accurate, and (iv) GPT-4 being accurate/faithful when giving descriptions of each feature. There is very little experimental evidence provided for confirming that any of these assumptions are true, and these claims are difficult to test in the first place. - In fact, the authors mention that a likely reason for the low correlation between the probe's predictions and the VADER lexicon (for some models) is "the complexity of the probe's task...a linear regression model is unlikely to recover such granular rewards accurately from just the activations" (L265-266). Although they do find a high correlation for one model, the insufficiency of this probe implies that it is not effective for accurately measuring the divergence between the model's activation patterns and the feedback label distribution. If the correlation is low, we cannot tell whether that is the probe's failure, or if the model has not acquired strong LFPs, or some combination of the two. Since this probing technique is a central contribution of the paper, I would expect stronger probes and more rigorous evaluation of the effectiveness of the probes. - How can one ensure that GPT-4's interpretations of the features are accurate or faithful? - Table 5 purports to check whether the predicted LFP-related features were actually important and useful to the LLM, but the numbers before and after ablation are often very close together (or identical, in the case of GPT-Neo-125m). It would be helpful to report confidence intervals or standard errors to check whether these differences are significant. But as it currently stands, this table's results does not seem to strongly support the claim that the predicted LFP-related features are indeed relevant to/critical for LFPs. - Lack of clarity in explaining methods: - Much of the writing about the methods is unclear, contradictory, or omits many details. For instance, the explanation of the logistic regression probe in L233-234 says "we label the concatenated activations as positive or negative based on the averaged activation deltas for each token over the entire input sequence, and train a logistic regression model to classify the activations," which would suggest that this probe's inputs are the activations. But L493 (in the appendix) says "...we give a positive or negative label to concatenated autoencoder outputs based on the sign of their activation delta. We then train a logistic regression model to predict the labels from the concatenated autoencoder outputs," which suggests that the inputs are actually the autoencoder outputs, not the original model's activations. Which is it? - In Section 3.4, how is GPT-4 prompted to provide the explanations? - Given how confusing and verbose the methodology is, I would encourage the authors to write out some of the procedures in equation form, rather than long paragraphs of text.

Questions

Questions are above.

Rating

4

Confidence

3

Soundness

1

Presentation

3

Contribution

2

Limitations

The limitations section was well-written and thorough, and covered many of the concerns I had myself. An additional limitation is that this method is computationally expensive and requires both training another model and running inference on a sufficiently powerful LLM (e.g. GPT-4) to interpret the features. In this paper, most of the results were for smaller models (under 1B params), and it is unclear whether this method would be scalable to larger models.

Summary

The paper investigates how large language models (LLMs) learn preferences from human feedback during fine-tuning using reinforcement learning (RLHF). The authors introduce the concept of Learned Feedback Patterns (LFPs) to describe activation patterns in LLMs that align with human feedback. They aim to measure the accuracy of these patterns in capturing human preferences by training probes on condensed representations of LLM activations. The probes predict the implicit feedback signal in these activations and compare it to true feedback.

Strengths

- The introduction of LFPs provides a new perspective on understanding how LLMs learn from human feedback. This concept helps in quantifying and interpreting the alignment between LLM activations and human preferences. - The authors validate their probes by comparing neural features correlated with positive feedback against GPT-4’s descriptions of relevant features. This cross-validation strengthens the reliability of their findings. - The use of synthetic datasets to elicit specific activation patterns in LLMs adds to the reproducibility and robustness of the study. These datasets are also made publicly available for further research.

Weaknesses

- The study primarily focuses on a few specific models (e.g., Pythia-70m, GPT-Neo-125m) and tasks (sentiment generation, toxicity), which might limit the generalizability of the findings across different LLMs and applications. More recently released models are of more value for studying RLHF patterns and verify that the method can be generalized. The patterns are easy to extract because that the used data are quite obvious to encode and decode. - While the probes show significant accuracy for certain tasks, the paper notes weaker correlations for more granular reward predictions, suggesting that the approach might struggle with highly detailed feedback signals. The issue of feature superposition in dense, high-dimensional activation spaces poses a challenge to fully interpreting the learned features. Although sparse autoencoders mitigate this to some extent, the problem remains a significant obstacle. - The validation process relies on GPT-4’s ability to describe neural features, which introduces a dependency on another model’s interpretability. This could introduce biases or inaccuracies if GPT-4’s descriptions are not perfectly reliable. - The paper acknowledges that while their method identifies features involved in feedback signals, it does not provide a mechanistic explanation of how these features interact or influence the expected feedback signal. This limits the depth of interpretability.

Questions

- Can you elaborate on how your findings can be practically applied to mitigate risks associated with LLM deployment, such as manipulation of user preferences or harmful behaviors? What strategies would you recommend for developers to monitor and adjust LFPs in deployed models? - Your validation relies on GPT-4’s feature descriptions. Have you explored other methods or models for validating the identified features? How do you ensure the robustness of these validations? - Have you tested your method on other LLM architectures or tasks beyond sentiment analysis and toxicity detection? If so, what were the results? How do you anticipate the effectiveness of your method would vary with different model sizes and types?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

discussed

Reply to rebuttal

I appreciate the reply from the author of this submission. I think all of them make sense and in all addressed my concerns. Looking forward to seeing progresses on the mentioned directions.