Hypothesis Testing the Circuit Hypothesis in LLMs

Summary

This paper considers the problem of formalizing and evaluating the circuit hypothesis. This hypothesis posits that specific subnetworks within a Transformer model are responsible for specific model behaviors. Although there are multiple examples of such circuits that have been manually discovered in the literature, the question of how to precisely judge whether a discovered circuit is indeed responsible for a specific behavior remains open. While past works have proposed various ad-hoc approaches, this paper first presents three criteria that an ideal circuit should satisfy (inspired by past works) along with corresponding statistical hypothesis tests for judging if a given circuit indeed satisfies these criteria. The paper then further presents weaker versions of the hypothesis tests for two out of the three criteria that are more likely to be satisfiable by circuits discovered in practice. These hypothesis tests are then applied to six different circuits that have been proposed in the literature to evaluate the extent to which they satisfy these criteria.

Strengths

This is a nice paper. It is well-written and addresses a question of growing importance. Mechanistic interpretability has emerged as a promising approach for understanding the behavior of trained LLMs. However, given the infancy of the area, the definitions and criteria for evaluating the quality of a mechanistic interpretation are yet to be standardized. This paper fills this need, particularly in the context of circuit analysis. The three proposed criteria of preservation, localization, and minimality make intuitive sense. The experiments with the synthetic circuits also provide evidence in favor of these criteria. I particularly like the proposed use of hypothesis tests to evaluate these criteria. These tests provide an operational definition of an ideal circuit and can be feasibly used in practice for judging new circuits. Overall, I think this paper helps bring some formal discipline to the topic of circuit analysis.

Weaknesses

I have a couple of concerns. First, the hypothesis test for judging equivalence (Equation 2) feels somewhat unintuitive. Consider the scenario where the circuit C* significantly outperforms the model M on half the inputs and vice versa. Intuitively, one would not consider such a circuit equivalent to the model, yet the hypothesis test would conclude otherwise. Why not define equivalence simply in terms of faithfulness of the circuit C* to model M? My second concern is the possibility that the three proposed criteria are insufficient for characterizing an ideal circuit, i.e., there might be additional criteria that are needed. However, given that circuit analysis is in its early days, I think the paper already makes useful contributions. I also have a number of minor comments and questions that I list below: 1. Line 180: Why use ; instead of , in s(C*(x);y)? 2. Equation 2: I think there is a missing abs operation. The definition of the null hypothesis in Appendix B.1 further suggests this. 3. Equation 2,3: Notationally, are lower case and upper case letters being used to represent separate things? For instance, x,y vs X,Y? Why not use lower case x and y in these equations? 4. Equation 5: In practice, there will be randomness also due to the fact that \delta(e,C) is calculated empirically. Does the test account for this randomness? 5. Line 267: The notion of a "reference distribution over circuits from the complement distribution" is not very clear at this stage. I understand what it means based on reading the rest of the paper. 6. Figure 3: The results for the IoITask are a little mysterious. How is the faithfulness of a circuit with some edges removed worse than an empty circuit? Further discussion would be helpful here. 7. Is Algorithm 2 an original contribution or was it presented in Gretton et al., 2007? Some more explanation and intuition about the algorithm would help the reader. 8. It would help to have a more careful comparison with the three criteria for circuits proposed by Wang et al, 2023.

Questions

Please address the questions listed in the previous section.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

4

Limitations

Yes, the paper discusses the limitations and the potential impacts of the proposed work.

Summary

This paper proposes a set of tests to evaluate how well a "circuit" meets its desired properties. - Here a circuit refers to a subnetwork, which could be either synthetic (e.g. constructed according to RASP) or discovered in a trained model. - The desired properties considered in this paper are: - *faithfulness*, where the circuit should preserve the performance of the original model; - *localization*, where removal of the circuit should alter the model's output; - *minimality*, which says the edges in a circuit (treated as a computational graph) should not be redundant. There are 3 "idealized tests" intended as necessary (but not sufficient) conditions: - **Equivalence**, where the null hypothesis is that the circuit $C^*$ preserves the performance of the original model $M$. - The test checks whether $C^*$ and $M$ have an equal chance of outperforming each other, which is a necessary condition (but not sufficient): rejecting the null means that $C^*$ necessarily does not preserve the performance. - **Independence**, where the null hypothesis is that the output of the model after removing $C^*$ is independent of the output of the original model. - The test computes the HSIC between the performance of $C^*$ and $M$. - Note that this is a stringent requirement; this will later be modified into a "flexible" test. - **Minimality**, where the null hypothesis is that all edges in $C^*$ are necessary and not redundant. - The test checks whether the amount of performance change by removing an edge in $C^*$ is more than the amount of change caused by removing an edge that is believed to be redundant. The paper also proposes 2 "flexible tests", where the difficulty of the test can be gradually varied by choosing different reference distributions when computing the test. - **Sufficiency**, which computes the probability (over random circuits in a reference distribution) that $C^*$ is more faithful than a random circuit. - **Partial necessity**, which checks whether with high probability, removing $C^*$ leads to a worse performance than removing a random circuit. The paper then applies these tests on 2 synthetic circuits and 4 circuits manually discovered from trained Transformers. - Synthetic circuits: for 2 types of the Tracr task, where the circuits are given as RASP programs. - Discovered circuits: Indirect object identification (IOI), Induction, Docstring (DS), Greater-Than (G-T). It finds that the synthetic circuits pass all 3 tests. In contrast, the discovered circuits fail the test to various extents, but are still far from random. Moreover, the flexible tests provide a more fine-grained understanding than the idealized tests (with binary outcomes).

Strengths

- The paper provides a quantitative way to evaluate the concept of "circuit" in models, by adopting the hypothesis testing framework. The proposed tests correspond to desired properties a circuit. - The proposed tests are more thorough and fine-grained than some existing evaluation criteria, such as the knockdown effect. - The paper are clearly written.

Weaknesses

- I'm concerned about the practical applicability of the tests. As the paper also mentioned, the proposed tests are either too stringent, or can be sensitive to the circuit size and the choice of the reference distribution. - The implications and use cases of the tests could be better discussed; e.g. please see questions below.

Questions

- About independency: neural networks are often overparameterized and hence likely contain a high level of redundancy. What if there are multiple circuits that are each faithful and minimal but are similar to each other? These seem to me should be considered as valid circuits, but they would violate independency. - About minimality, I don't see why the randomly added edge would lead to a small $\delta(e^I, C^I)$: it's possible that adding a random connection would change the performance non-trivially (but likely negatively). For example, even a simple residual term would change the scale of the output and affect subsequent computations. - Could you comment on the effect of different choices of activation patching? - Could you comment on how the proposed tests can inform interventions on training (e.g. as regularization terms)? The hope is that this could make the trained networks more likely to contain circuits that more closely satisfy the desiderata. - Can the proposed tests help with coming up a more precise definition of circuits, or inform how we should choose the granularity of the definition of "nodes"? Minor clarifications: - Eq (2): should the LHS be taking an absolute value? - Line 246: not sure I understand what “a lucky draw” means; e.g. it could mean a draw that is better than q* fraction of random circuits? - Line 250: what does "supersets of C" and "comparable to C" mean?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper discusses the technical limitations. There is no direct societal implication.

Summary

The paper operates in the framework of mechanistic interpretability of transformer models, where it is assumed that subgraphs (i.e. circuits) of the computational graph determined by the model implement specific capabilities of the latter. It defines a set of properties that an ideal circuit should have, namely equivalence, independence and minimality. Afterwards, it proposes two sets of hypothesis tests. The first set identifies a more strict hypothesis-testing framework to determine if a circuit satisfies these properties. The second proposes some more flexible tests for the first two properties of ideal circuits. Afterwards, these hypothesis tests are used on synthetic and manually discovered circuits in the literature on GPT-2 small and other small transformers.

Strengths

- Overall very well-explained paper, particularly the part on mechanistic interpretation. - Crafting of the hypotheses for testing is original and well thought. It follows logically from the properties defined and is well explained. - Different types of tests were performed with different granularity.

Weaknesses

- I believe it would have been nice to perform further tests to understand whether the idealized tests have some applicability on discovered circuits (or if they always lead to the null being rejected), potentially by also altering the discovered ones. - Would have found it interesting to propose some ideas on how to use these tests for circuit discovery.

Questions

- I believe there is a typo in the description of Table 1 on page 7, where it is claimed that “A (✓) indicates the null hypothesis is rejected”, while I believe that this means that the circuit “passed the test” so that the null-hypothesis is not rejected. - When referring to the Bonferroni correction, was the base one used or the Bonferroni-Holm one? The latter is always more appropriate as it is less conservative and leads to a more powerful test.

Rating

6

Confidence

3

Soundness

2

Presentation

4

Contribution

2

Limitations

-