Summary
This paper proposes an approach for the automatic discovery of circuits (ACDC) in artificial neural networks (applied to transformer-based LLMs), which works by recursively constructing a subgraph of "important" nodes identified through the patching of model activations on datapoints relevant to a specific task (the choice of which is, in general, non-trivial). The authors demonstrate, through many experiments, that ACDC is mostly able to faithfully recover circuits which were manually identified by previous researchers on a variety of tasks (notably Python docstrings, IOI and induction heads), thus automating a highly labor-intensive part of the circuit discovery process. Additionally, they explore choices of patching value, target metric and threshold value, whilst also demonstrating that other comparable methods for distillation/subgraph isolation are not as well-behaved as ACDC.
In addition to the paper, the authors also release an open-source implementation of ACDC which has already been applied to some success by other mechanistic interpretability researchers already.
Strengths
The methodology for automatic circuit discovery proposed by the paper extends previous approaches for activation patching to automate otherwise labor-intensive mechanistic interpretability work. This in of itself is not a significant novelty, but the paper's strength lies in a thorough experimental investigation of the benefits of ACDC over other subgraph discovery methods, coupled with new methodological insights on how best to perform activation patching. In particular, they provide two novel findings: 1) KL Divergence is more well-behaved than logit differences when performing activation patching for circuit discovery, and 2) Zero patching, whilst significantly OOD, is often more effective than patching corrupted activations.
The presentation of the paper is very good, with a coherent narrative for the experimental investigations and clear figures supporting all claims. Additionally, the supplementary materials provide further interesting discussion and results, which given the exploratory nature of circuit discovery is highly valuable.
Lastly, the release of the accompanying ACDC algorithm for use by the community is a significant contribution in of itself, as demonstrated by the fact that other members of the MechInt community have already applied ACDC for their own research.
Weaknesses
No major weaknesses were identified.
There are a fair number of minor phrasing issues outlined in the following nitpicks section. In the related work section, explicitly stating how path patching varies procedurally from ACDC might be worthwhile. Additionally, a discussion of how ACDC varies from Causal Scrubbing in its patching methodology may be useful.
## Nitpicks
* 3: makes it **too** costly"
* 27: circuits **as** subgraphs
* 32: "with which to extract" or "for extracting"
* 32: remove "that automates part of it"
* 85: The choice of phrasing - "clearly defined behaviour" makes this sentence almost tautological. Perhaps an explicit mention of simplicity would be suitable here. Researchers unfamiliar with MechInt may consider e.g. "Writing python code" clearly defined or "writing python docstrings" too broadly defined.
* 91: "**Tasks** 1 and 3" inside the parentheses
* 94: from **each task, which** researchers
* 98: **as** a computational graph
* 101: "on the level of detail of _their_ explanations of model behaviour* subject unclear and wording confusing
* 125: such as -> for example (as subject of "such as" could be "tasks"). "predict correct gender predictions" remove last predictions?
* 223: It seems the discussion of zero ablations takes place in Section 5 and Appendices F.2, rather than Appendix D.
* 233: "we explain how compare to" -> "we compare to"
* 236: "experiments use the same modifications to SP and HISP *as*"
* 308 "is known*, and through comparison with previous*..."
* 310: not clear what "outline of a subgraph" means vs. just "a subgraph"
* 318: "work; a novel contribution" or "work - a novel contribution".
* 320: "*within* the community"
Questions
The following questions are not crucial to the narrative of the paper, nor potential critiques of completeness.
1. Have you thought of automated ways to trade-off circuit faithfulness versus sparsity, without re-running ACDC with different values of $\tau$?
2. In general, how sensitive is ACDC to the choice of clean and corrupted datapoints. E.g. for the IOI task does ACDC provide considerably different circuits if very few examples are provided, vs. many? What about paraphrasing or noise injection as in ROME (this is probably very task dependent)?
3. When modifying Subnetwork Pruning you discuss interpolating the mask values - should we expect that linearly interpolating between a clean and corrupted activation is principled (i.e. does not potentially shift the representation to something meaningful but distinct)?
4. Are all values of SP masks 0, or 1 by the end of SP training? If not, then for the sake of counting subgraph edges, what is considered an "unmasked node"?
5. Given the unexpected utility of using zero ablations, would the authors suggest trying this whenever utilizing ACDC?
6. Could the authors expand on how the _locally significant changes_ alternative to detecting salient parts of the subgraph would avoid potential sensitivity to the form of patching? Presumably, some perturbation would be required to measure "effects" and it is not immediately clear what this perturbation is, if not an activation patch.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Limitations
Limitations are addressed, or explored through supplemental experiments.