InversionView: A General-Purpose Method for Reading Information from Neural Activations

Summary

The paper introduces the method InversionView, which finds inputs that give rise to similar activations (the preimage). As the preimage grows exponentially with sequence length the authors train a conditional decoder to generate inputs from a target activation. The authors show three use cases: a character counting task, Indirect Object Identification, and 3-digit addition.

Strengths

The approach of generating text examples that lead to similar activation patterns is intuitive and could be useful for interpretability since results are very human readable. The example use cases (especially the toy task of character couting) seem promising.

Weaknesses

- the abstract could be a bit more specific - in general, I think the reader is left wondering for a bit too long what the authors actually do and how the method works. - line 28: "which in turn helps us put together the algorithm implemented by the model." promises a bit too much - motivation and description of the method is too vague - very few details on the method/decoder that generates the examples that are later inspected - results of experiments are a bit hard to pass. It would be nice if the reader is guided through those a bit better at different levels of abstraction - the approach (generating human interpretable examples that achieve a specific activation patters) seems closely related to feature visualization/ adversarial example and counterfactual example generation/ in general methods that generate in the input space, however those works are not discussed at all

Questions

- a graphic showing how to get from source transformer activations to pre image samples would be helpful - Figure 2 is hard to parse: I would suggest highlighting your conclusion/interpretation (aka one activation encodes target character but not count, one activation encodes count but not target character etc) in the figure. - in general, for the experiments, I would suggest to clearly state the task and your findings before jumping into technical details - 3.3 "We apply InversionView to the components of the circuit" seems like you apply inversionView to specific activations found by [39] but then you state "InversionView unveils the information contained in the activation sites, with results agreeing with those of Wang et al. [39], while avoiding the need for tailored methods." So it seems like you needed the "tailored methods" to know where to apply inversionView in the first place, or am I misunderstanding sth?

Rating

4

Confidence

3

Soundness

2

Presentation

2

Contribution

3

Limitations

- I think when moving from toy tasks to SOTA models, inversionView might become very prone to human tendency for pattern matching - in addition polysemanticity might be very relevant for InversionView results but is not discussed at all - related work is not discussed in sufficient detail

Summary

The method proposed in the paper seeks to decipher the information encoded within neural network activations. The core idea is to examine subsets of inputs that produce similar activations, utilizing a trained decoder model conditioned on these activations. The authors perform their analysis on three different tasks: character counting, indirect object identification, and 3-digit addition.

Strengths

* The proposed method helps us find which parts of a network handle specific tasks. For instance, in a counting task, we can see which activations detect the target character and which ones do the counting. This is interesting because it helps us understand how neural networks work inside. * Showing the underlying algorithm the model uses to solve tasks, like adding 3-digit numbers, gives useful insights.

Weaknesses

* Applying the method to extremely large models and more complex tasks is challenging. * Selecting an appropriate distance metric and epsilon is challenging and requires tuning, making it difficult to adapt the method to different tasks. This also highlights the authors' impressive effort in making the method work.

Questions

* Do you think it is possible to use other distance metrics besides L2-based metrics? Additionally, why do you believe the L2 distance is effective in this context? Is there something unique about the geometric space of these tasks that makes L2 distance particularly suitable? What types of tasks, in general, can be effectively solved using L2-based metrics?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

This work is mainly based on the representational geometry of activations in the activation space and chooses samples whose distances are within a defined $\epsilon$-preimage distance. A single two-layer transformer decoder model is trained using activations of each layer from the investigated model. After finishing decoder training, the Euclidean distance is used as a metric to select generated samples based on each activation presentation of each layer, and compare those samples with the query input sentence. This approach is evaluated and analysed under three tasks, i.e., character counting with a 2-layer and 1-head transformer, IOI with GPT2-small and 3-digit addition with 2-layer, 4-attention head transformer. The comprehensive experiment results and analysis confirm the geometry hypothesis, and provide lots of insights to future work.

Strengths

- Activation representation of each layer from small neural networks to GPT2-small can be visualized and explained based on the geometry hypothesis with a trained decoder using those activations as input - This hypothesis is evaluated using different tasks, i.e., character counting, IOI and 3-digit addition, and lots of meaningful analysis and insights are discussed - Experiments are solid and the appendix includes many details about each case study.

Weaknesses

- I suggest authors use another figure to demonstrate the whole training and evaluation pipeline, i.e., what input is used for training probed model, what query input is for decoder training and what input and output are from the trained decoder, which will help to understand the whole framework much easier for readers. - This method is mainly based on manual investigation to select and analyse generated samples, which might need large labour resources to extend to large-scale LLMs.

Questions

- In line 115, what does $\mathbf{z}$ represent here? and what is the difference between $\mathbf{z}$ and $\mathbf{z}^q$? - In line 108, Is this input the same as the query input in figure 1? Another figure introducing the whole pipeline might help a lot to understand. - Does this neighbour of geometry hypothesis always hold across all LLM activations?

Rating

7

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

I am looking forward to how to automate this approach and apply it to the larger LLM interpretability analysis.

Summary

In this paper, the authors propose InversionView, a method to inspect the information encoded in neural activations. The proposed method is based on checking the activations difference given different inputs. The authors showcase the effectiveness of this tool on mainly three tasks: character counting, indirect object identification, and 3-digit addition. Strengths 1. The topic of model interpretation is important. 2. The empirical results confirm the effectiveness of the proposed method. 3. The paper is well written and easy to follow. In addition, the authors provide abundant experiments to showcase the effectiveness. Weaknesses 1. Although being described through text, the precise algorithm for the proposed method is unclear to me. 2. It is unknown how the proposed method scales with larger models. The largest model size in the paper is limited to GPT-2, which is not considered large nowadays. Overall, I think the paper is interesting and recommend a weak acceptance. I will consider raising my ratings if the above weaknesses are addressed.

Strengths

1. The topic of model interpretation is important. 2. The empirical results confirm the effectiveness of the proposed method. 3. The paper is well written and easy to follow. In addition, the authors provide abundant experiments to showcase the effectiveness.

Weaknesses

1. Although being described through text, the precise algorithm for the proposed method is unclear to me. 2. It is unknown how the proposed method scales with larger models. The largest model size in the paper is limited to GPT-2, which is not considered large nowadays.

Questions

See the weaknesses.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have adequately addressed the limitations.