Summary
The authors investigate the potential issue of identifiability in mechanistic interpretability through experiemnts on small MLPs where (isolated) circuits are ennumerated and assessed fairly exahustively. They find identifiability is an issue at all levels: in the number of subcircuits functionally aligned with the full network, in the number of algorithms consistent with the behavior, and in the mappings between algorithms and circuits. This problem gets worse as architecture size increases, and training on a greater number of tasks only mitigates this issue to some extent.
Strengths
* Well written
* Well referenced
* Adresses an issue of interest to the interpretability community
* Provides exhaustive experiments with well-understood ground truth.
* Investigates the effects of various variables (architecture size, number of tasks, noise) on the identifiability problem.
This kind of study is very much needed.
Weaknesses
I first summarize the most important points and then elaborate.
* It is unclear whether the conclusions about identifiability problems can be stated generally or only in particular for circuits as isolated objects (circuit discovery through zero-ablation), which might mischaracterize the functioning of the network as a whole.
* The possible improvements advertised in the abstract/introduction are rather only sketched in section 5.1.
Is the large number of circuits/possible explanations due to looking for circuits in isolation (e.g., via zero ablation) rather than working in context with the rest of the network (e.g., via activation patching)?
Line 320 describes the circuit isolation procedure. This is equivalent to zero-ablation and the criterion is equivalent to the definition of suficcient circuit.
How would identifiability look like if we chose to define circuits as they function in the context of the full network? See for example the definition of circuit via activation patching in Adolfi et al., 2024.
Isn’t it possible that many of the isolated circuits discovered through zero-ablation are mischaracterizations of the in-context functioning of the circuits as they are embedded in the full network?
Line 080: “a model’s behavior should have a single, well-defined explanation”. There is no citation here and it is unclear where this intuition is coming from, what is it’s theoretical support, etc. To offer a counter-intuition: consider a circuit that is sufficient on its own to mimic the behavior of the full network over some input domain; such a circuit need not be unique. Trivially, the circuit plus additional neurons to form the full network is another such circuit. But there is no contradiction in intuiting that multiple such circuits of different sizes, with partial or no overlap exist in the network and, in principle, offer alternative (perhaps incompatible?) ‘explanations’ (see Adolfi et al. 2024 for theoretical analyses).
On Line 091: the authors mention “the identifiability properties of current MI criteria”. The criteria of interest that define circuits leave open the possibility that these circuits are not unique. So the definition of these circuits does not preclude non-identifiability unless the uniqueness property is trivially appended to the definition. This leads one to suppose that uniqueness under the typical definition of circuits is a property left to be determined empirically. It could, in principle, be motivated theoretically, but I see nothing in that direction here. Is it possible to provide some theoretical motivation for uniqueness that is not trivially stipulated but justified from first principles?
If a network implements the same functionality for different inputs through different circuits and algorithms, does this really make mechanistic interpretation hopeless? (i.e., in this case, is only a functional explanation capable of unifying all the existing ‘explanations’?). It would be useful to have any assumptions about satisfactory explanations made explicit in the manuscript.
Line 489: “the challenge lies in defining criteria that distinguish valid explanations from misleading ones.”
It seems to me that, conceptually, identifiability does not pose a problem for distinguishing misleading from valid explanations. The problem arises only if an explanation is presented as unique or valid for a full input domain when this is not so. This issue might warrant some clarification.
Line 490: “According to MI, the explanatory algorithm should be unique, meaning multiple competing explanations should not exist.” But this statement is made without citation. This assumption seems ill-founded to begin with, for the reasons mentioned above. Where does the criterion come from?
Questions
Minor comments, questions, and suggestions:
Line 403 states that “larger architecture…could also lead to greater overparameterization”. This could benefit from elaboration; in particular, how larger architecture could lead to a reduction in the number of valid abstractions.
On Line 065, “Given the near impossibility of exhaustively searching all possible algorithms across all subsets of a neural network”, I might suggest to reframe this not as impossibility but as intractability, infeasibility or implausibility. Certain interpretability queries might have large search spaces that could nevertheless be searched efficiently. The relevant property is the complexity of the interpretability query, not merely the size of the search space. For computational complexity analyses of circuit discovery, see Adolfi et al 2024.
On Line 067, the authors state “researchers have developed approximation methods with different assumptions and trade-offs”. It seems to me that the circuit discovery methods that are typically developed are heuristics for circuit finding, not approximation algorithms with any proven guarantees. In any, case it would be useful if the authors can distinguish between these two categories in their descriptions.
Citation to Van Rooij on Line 046 does not seem to fit with the corresponding sentence, as that paper does not deal at all with interpretability, as opposed to Lindsay, 2024, which is indeed an approapriate citation. For examples of studying the fundamental properties of (inner) interpretability queries see Adolfi et al., 2024.
Section 2.1 mentions interpretability work on transformer models but only in language. An example from vision transformers can be found in Vilas et al. 2023.
Please clarify the notation in Definition 4.
Line 229 makes an implicit statement about computational complexity but provides no citation. See Adolfi et al. 2024 for relevant complexity analyses. This is also relevant to the statement on Line 257. Here it would also be useful to clarify how uniform random sampling “approximates” the desired measure, as this seems non-obvious. Perhaps the authors mean random sampling is a heuristic with unknown properties?
Line 494 states that current MI methods can only approximate their targets because exhaustive enumeration is impossible for large models. This is technically incorrect, as even for some problems with exponential serach spaces, efficient search algorithms that find optimal solutions are possible. The relevant notion is the computational complexity of the interpretabililty queries, not simply the size of their search space (see Adolfi et al., 2024).
Section 2.1 describes a parallel between AI interpretability and neuroscience. A framework that draws from lessons grounded in this parallel is described in Vilas et al. 2024. This framework provides a nice embedding for the what-where distinction, corresponding to the algorithmic and implementational levels, respectively.
The problem of identifiability interacts in interesting ways with the computational complexity of circuit finding. Adolfi et al. 2024 analyses circuit queries that are relevant to the authors’ points on identifiability. See, for instance, counting problems which ask for the number of circuits in a train neural network that have a certain property (e.g., they are sufficient for a behavior). Furthermore, if the number of sufficient circuits is typically large, heuristics for otherwise intractable problems (e.g., sufficient circuit) could seemingly find their targets in a feasible amount of time. In this scenario, non-identifiability is an important catch.