Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms

Summary

Edge attribution patching (EAP) is a circuit discovery technique approximating the impact of causal interventions on model edges via gradient attribution. Its effectiveness is commonly validated by comparing the overlap of its resulting circuits with those found by activation patching methods with causal guarantees, but rarely in terms of faithfulness, i.e. whether behavior of the model remains consistent after all non-circuit edges are ablated. This work proposes an extension to edge attribution patching, EAP-IG, using integrated gradients to improve the faithfulness of the resulting circuits. The faithfulness EAP, EAP-IG and activation patching is evaluated, showing that no-overlap and full-overlap between EAP-like methods and activation patching are generally good indicators of unfaithful and faithful (respectively) circuit identification. On the contrary, circuits with moderate overlap cannot generally assumed to be faithful to model behavior, pointing to the importance of faithfulness evaluation in this setting.

Rating

8

Confidence

5

Ethics flag

1

Reasons to accept

- The proposed method and related analysis in the paper are explained thoroughly and suggest a promising way forward in the evaluation of circuit discovery approaches. - The analysis of disagreement between overlap and faithfulness metrics of circuit quality provides a valuable perspective, identifying existing limitations in current evaluation practices for attribution patching approximations. - The proposed EAP-IG is a refreshing application of an established feature attribution method for model component attribution. Notably, it allows for different gradient propagation targets and is convincingly shown to be more faithful than EAP. - An evaluation is conducted to ensure the findings generalize across several tasks and model sizes.

Reasons to reject

- Minor: the name of the model evaluated in the main body of the paper is never mentioned explicitly, although I assume it is GPT-2 small based on the task section. - While results discussed in Appendix C suggest that a low number of interpolation steps might be sufficient to obtain good faithfulness performances, this estimate is based on a set of toy tasks. There is no guarantee that such property would hold in case of more complex tasks. - More generally, as for other circuits discovery approaching requiring corrupted examples, EAP-IG is limited by the need of such pre-defined pairs to function. Provided that several tasks among those used here admit several possible corrupted variants, ideally a broader set of baselines should be evaluated for every example, and confidence intervals should be reported to confirm the robustness of the proposed method.

Summary

The paper introduces a new method called Edge Attribution Patching with Integrated Gradients (EAP-IG), which aims to efficiently identify circuits in language models while ensuring these circuits are faithful. EAP-IG, which combines EAP with IG to better assess edge importance in circuits, especially addressing the issue of zero gradients which can misrepresent an edge's importance by extrapolating effects between two points using a linear expansion. The main results show that circuits found using EAP-IG yield with high faithfulness as defined in the paper. Additionally, the authors also explore the circuit overlap between tasks.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

- The research topic, to faithfully scale circuit discovery method, is very timely. The paper extends previous circuits discovery methods EAP by taking inspirations from IG. The proposed method EAP-IG better assesses edge importance in circuits, especially addressing the issue of zero gradients which can misrepresent an edge's importance by extrapolating effects between two points using a linear expansion. - The paper is easy to follow, and study the topic both theoretically and also empirically. The authors also show circuits similarity between tasks, which is quite novel. The paper found that similar tasks often come with more overlapping circuit components.

Reasons to reject

- The definition of faithfulness could be more precisely grounded. The faithfulness described in the paper refers specifically to whether the identified circuit comprises a complete set of components that control model behaviors, **conditioned on** zero-ablate or mean-ablate interventions. However, this circuit might not be the definitive one. Furthermore, iiuc, there is a potential "hydra-effect," where multiple circuits exist, and one circuit may only become apparent due to interventions. - Section 4.4 could be clearer. I am not an expert on this, but it feels somewhat counterintuitive to see that vanilla activation patching could outperform gradient-based methods in terms of improving faithfulness—that is, the completeness of the identified circuits with respect to task performance. The analyses of overlap between different patching methods (gradient-based or intervention-based) might also suggest that numerous types of circuits exist, which makes the claim that any one of them is more faithful seem unfounded. Could the authors clarify this?

Summary

This paper presents a simple adaptation to edge attribution patching (EAP) for scoring edges of circuits in an LLM by borrowing the notion of integrated gradients by Sundararajan et al. 2017; doing so allows for the extraction of more faithful circuits when compared to EAP performing on a par or even better that the more expensive non-approximate method of activation patching circuits. The authors also do a study on the correlation between overlap of circuits (nodes, edges) and the faithfulness which doesn't necessarily lead to a causal relationship. The evaluation is done using GPT-2 small (although the appendix has a couple more experiments using GPT-2 XXL and Pythia-3.8B) on 6 corruption tasks.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

* The contribution of this paper is not so much on the novelty of the proposed algorithm but on the simplicity, elegance and effectiveness of it. Just to be clear, I am not suggesting that the produced algorithm isn't novel (on the contrary!) but that the outcome essentially stems from an incremental observation of previous works. Having said that I believe that since there is a substantial improvement wrt faithfulness it's an addition to the mechanistic interpretability community that will be received positively. * I believe that the more significant contribution stems from the extensive analysis and experiments conducted to verify that EAP-IG works and why. The overlap vs faithfulness findings were also interesting and I appreciated the pragmatic and somewhat cautious view that overlap isn't always a good predictor of faithfulness across tasks. * For someone who is not an expert on mechanistic interpretability, Section 2 is a very nicely written introduction to the field; much appreciated!

Reasons to reject

* I found the extra Greater-Than tasks that were constructed to verify the existence of robust circuits that appear across tasks to be somewhat obvious and derived from the simplest of all tasks in my opinion. Personally, I found the relationship between hypernymy and the country-capital tasks a lot more interesting (and fascinating!), so I wonder why the authors didn't pursue constructing similar derivative tasks with sub-categories that have expert-defined similarities, e.g., countries that are nearby, hypernyms that are farther/closer to each other according to Wordnet etc.