Obtaining human-interpretable explanations of large, general-purpose language models is an urgent goal for AI safety. However, it is just as important that our interpretability methods are faithful to the causal dynamics underlying model behavior and able to robustly generalize to unseen inputs. Distributed Alignment Search (DAS) is a powerful gradient descent method grounded in a theory of causal abstraction that has uncovered perfect alignments between interpretable symbolic algorithms and small deep learning models fine-tuned for specific tasks. In the present paper, we scale DAS significantly by replacing the remaining brute-force search steps with learned parameters -- an approach we call Boundless DAS. This enables us to efficiently search for interpretable causal structure in large language models while they follow instructions. We apply Boundless DAS to the Alpaca model (7B parameters), which, off the shelf, solves a simple numerical reasoning problem. With Boundless DAS, we discover that Alpaca does this by implementing a causal model with two interpretable boolean variables. Furthermore, we find that the alignment of neural representations with these variables is robust to changes in inputs and instructions. These findings mark a first step toward faithfully understanding the inner-workings of our ever-growing and most widely deployed language models. Our tool is extensible to larger LLMs and is released publicly at `https://github.com/stanfordnlp/pyvene`.
Paper
Similar papers
Peer review
Summary
This paper introduces an advancement of Distributed Alignment Search (DAS), termed Boundless DAS, where the brute-force search steps are replaced with learned parameters. This enhancement enables efficient exploration of interpretable causal structures in large language models and therefore results in a scalable interpretable model.
Strengths
1) The increasing significance of LLMs necessitates an evaluation of their interpretability. Developing a scalable method to achieve this goal is undeniably crucial and intriguing. 2) Current interpretability methods are inadequate for LLMs due to scalability issues, rendering them impractical. In contrast, the proposed method offers a solution that enables the interpretation of real-world LLMs, filling this crucial gap.
Weaknesses
1) My primary concern lies with the novelty of the paper, which appears to be exceedingly limited. The paper seems to be primarily an extension of DAS and a direct amalgamation of two existing works, namely DAS and neural PDE [58], without addressing any specific challenges or introducing significant innovative elements. 2) The paper heavily relies on the DAS method as its foundation, and it is imperative to provide a comprehensive explanation of DAS, including its limitations, which are currently absent in the paper. A thorough understanding of DAS and its shortcomings is crucial to fully comprehend the context and motivation of the proposed research.
Questions
Please refer to Weaknesses.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
2 fair
Contribution
1 poor
Limitations
The authors have discussed the limitations of their work.
Summary
Finding alignments between variables in a user-hypothesized causal model for how a task could (or should) be performed and representations in neural networks, termed "causal abstraction", is a mechanistic interpretability technique for understanding how neural networks develop predictions at a granular level by using causal interventions. Prior work has recently proposed the Distributed Alignment Search algorithm for efficiently and robustly finding this alignment, but it still relies on brute-force search over the dimensionality of neural representations. This paper proposes to circumvent the brute force search by *learning* parameters of an orthogonal rotation matrix to maximize the alignment between a neural network representation's linear subspace and one of the variables in the high-level causal model. They test their method on one rather narrowly scoped task, which is determining if a price is between two numbers, and use the Alpaca (instruction-tuned LLaMA) 7B decoder-only language model. The authors also test whether the found alignments generalize across settings (both input prompts and output token choices for labels), and find that they do. Edit: I read the authors' rebuttal and the other reviews & discussions. I did not change my score as only one of the weaknesses (writing/presentation) was alleviated. I still think the paper should be accepted though.
Strengths
Originality: - The application of causal abstraction (or even other variants, like causal mediation analysis/causal tracing) to large instruction-tuned LMs has not been done; this is highly original to the best of my knowledge. Quality & Clarity: - The experiments seem sound. The analysis in sections 4.3 - 4.6 is insightful and interesting. The overall findings of the paper are useful to a larger community for understanding math processing in LMs. - The writing is clear overall. - Related works section is comprehensive. Significance: - Causal analysis of language models at the representation-level is a very important goal with potentially far-reaching impacts for the broader NLP community. Using instruction-tuned models that are in widespread use today, such as Alpaca 7B, bridges the gap between theory and practice and makes this work more applicable to a wider audience.
Weaknesses
- The paper does not read independently (particularly section 3.1) to a reader unfamiliar with prior work. I had to refer extensively to [32] to grasp what the various terms and variables used meant, and to understand what components of the proposed algorithm are novel w.r.t prior work. If there is not space in the main paper, the appendix should be used to thoroughly explain these concepts in a camera-ready version. The math notation could be improved; see "questions" section. - The method involves pre-registering some possible symbolic models for performing a task; it cannot discover novel symbolic models that are not specified by the user. - The method is tested on one narrowly-scoped task (determining if a price is between two numbers) and one model. There is no way to "validate" a mechanistic interpretability method beyond principle or axioms as we don't have the ground truth, so some of the findings are more suggestive rather than extremely concrete conclusions. However, this is a common problem in the subfield and I think this paper is convincing overall.
Questions
- On finding strong alignments between Alpaca’s representations and two of the proposed causal models ("left boundary","left and right boundary")— if the method reveals a faithful abstraction, wouldn't that imply that only one abstraction can have strong alignments with model weights, rather than multiple? Can you explain? - On writing clarification in Section 3.1-- it would be useful to establish early on that "variables" refers to hidden representations in a NN and "algorithm" to a high-level user-specified symbolic model. Relatedly, the terms "input variables" and "target variables" are a bit confusing as you also have input sequences ("base input", "source inputs") and target model outputs, which are different. - I’m not sure I would use the term “largest” to describe a 7B model. - You use the phrase “human-interpretable” to describe the explanations that DAS produces; it would be good to provide a definition or at least some citations of what this concretely means. - Lines 170-171 and x-axis in Figure 4: these refer to the same thing; it would be good to give a concrete in-line example in the text as you have done for the core instruction. - Will you open-source your code? Some other mathematical notation could be improved. For example, - what do $j$ and $k$ represent? (I'm assuming $k$ is the number of dataset instances and $j$ the instance being indexed, but this is unstated). $k$ appears doubly-defined in Eqn. 3. - Set notation is mixed between uppercase variables and brackets - $\mathbf{s}$ and $\mathbf{b}$ are not vectors but sequences of inputs (i.e., tokens), which is a bit nontraditional notation; introduction of $\textbf{Inputs}$ seems as though it should have been introduced earlier. - both $\mathcal{N}$ and $\mathcal{M}$ represent a neural network model, both "algorithm $\mathcal{A}$" and "model $\mathcal{C}$" (which is never used after defining) represent a high-level causal model, if I understand correctly. - no definition of $F$, $F^*$, $\Pi$, $L$ - I did not follow the connection between target variables $X$, $\mathbf{Z}$, and $\mathbf{N}$. - $y_p$ and $y$ are more traditionally notated as $\hat y$ and $y^*$; and should keep indexing consistent as $\hat y_i$ and $y^*_i$
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
4 excellent
Limitations
Yes- sufficiently detailed & comprehensive limitations section.
Summary
This paper builds on Distributed Alignment Search by introducing a more efficient learned model to search for a given circuit in a model. They test this method on a specific task: asking the model whether a given numeral falls between two other given numerals. They have a single task, and find an alignment with 2 possible circuits for accomplishing this task and fail to find a good alignment to 2 other handcrafted circuits, demonstrating that this method may be capable of distinguishing between correct and incorrect circuits. EDITED to raise soundness score.
Strengths
The approach of boundless DAS itself seems to be a worthwhile one, and I believe that it could work. The approach scales well, up to Alpaca, and therefore could be very useful in modern models. I'm impressed by the approach of carefully constructing possible circuits to test, and it seems to work surprisingly well in LLM settings. The observation that the IIA patterns are more structured for the seemingly correct circuits is an interesting one, though I have some reservations about this as a verification method (in questions).
Weaknesses
An obvious criticism of the method itself is that we lose a degree of interpretability by introducing learned parameters into the interpretation method. How many learned parameters does it take before you’ve just introduced another blackbox as your lens? If the problem is that you don’t trust learned parameters to behave interpretable, how can you ensure that you are behaving interpretably? In particular, it doesn’t seem to get away from the complaint that the authors themselves introduce: failing to find an alignment does not mean that the alignment doesn’t exist. This is a mild criticism, and I still consider the method to be potentially useful. More significantly, the experiments are not convincing. There is only a single task evaluated. They consider only a single model, Alpaca. There are only four possible circuits considered. All of these factors add up to a fairly unconvincing set of experiments which have limited statistical power. Outside of the main experiments, these issues become more apparent: section 4.5 tests generalization from a fixed input to a different fixed input, with only a single sample considered for each case. A test of generalization with inserted context only considers two different possible contexts. Overall, these experiments are too constrained with too few samples to be convincing; while I don't think that this is cherry picked, results that comprise of case study of only a couple of samples could easily lead to publication bias. Furthermore, for the price task, the effectiveness of the alignment is measured by IIA, but the baseline for IIA is based on an assumption of complete randomness that I do not think is valid. After all, we have a method that has already found an alignment that maximizes the match with the circuit. I would trust this more if you had, e.g., tested it on random weights or weights trained for a completely different structured task that wasn't actually language modeling. On another note, 4.5 appears to be the method that they should be using as the main experiment. I don't trust results that train the boundary parameters and alignments on the same inputs that they test on, which appears to be the approach taken in 4.3. The math could be more carefully explained, with more precise definitions for the variables involved and more detailed intuitions for the method. One way to help is to turn Figure 1 into a step-by-step algorithm diagram. Minor: - "Boundary" is an overloaded term (used in both the method and the task). You should pick different wording. - "We find that Alpaca achieves near-perfect task performance because it implements a simple algorithm with interpretable variables." *Because* is a *very* strong claim. Do they actually support the idea that the simple algorithm is the reason for the task performance? Is performance worse for tasks that don’t have simple algorithms? They do not show either piece of evidence.
Questions
Why would baseline IIA be 50%? You are actively aligning a circuit to the model, so shouldn't you be able to find an alignment that performs above 50% even in a randomly weighted model? What is a target variable $Z_j$? What if $F_N$? Please explain the boundary index variable in more detail. When you restrict $b_j$ to be "a multiple of $b_{j-1}$ with a factor of 2", do you mean that $b_j = 2 b_{j-1}$ or that it could be any multiple as long as it is also even? Why would you use the notation in this way, rather than specify the variable if it is deterministic? Is it likely that we could have a very high IIA score without having more structured patterns? Rather than validating the IIA score, the observation of sparser patterns seem to strongly correlated with the score by definition, so I don't entirely trust it as a verification of the faithfulness of the alignment. What is an intermediate boolean value? How large is the task dataset used to learn the alignment? Is it different from the set used to test IIA for the alignment? "We sample 100 experiment runs from our experiment pool and create two groups" Do the runs differ only by the input used? Or by some random seed? In section 4.6, what does it mean to look at learned boundary width? What does a higher width indicate? Are the training steps (Figure 5 x axis) training steps for learning the alignment, or for the training of the LM? The number of steps is small so I assume it is for the alignment. What do we learn from the learning curve, that we didn't get from the final values?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
2 fair
Contribution
3 good
Limitations
The authors acknowledge that their method is limited by the fact that not finding an alignment is not evidence that there is no alignment to the given circuit, which is very good. However, the limitations of the experiments themselves are not acknowledged. This method only applies to synthetic tasks where we know the possible algorithms. This isn't explicitly acknowledged.
general comment
I think that the approach here is potentially very valuable, but this paper remain a case study of a single task, and the evidence is extremely limited for the reasons I've given. It provides a single very limited example without any further validation of the method, e.g., by showing that correcting the circuit found can fix an incorrect judgment.
New Baseline with Randomly Initialized LLM
Thank you for your continued engagement. We were able to pursue a version of the random LM experiment you suggested, and this proved fruitful. We randomly initialized a model, which resulted in near 0% task performance on the price checking task, as expected. After running boundless DAS on first layer of the random LLM, the alignments to each token's residual stream ranged from 0 % to 69% IIA, which is comparable to a most frequent label dummy model (66%). For the representations with near chance IIA, this means that we were able to find a distributed neural representation that shifts the probability mass to "yes" or "no", but does not differentiate between the two. We also found that we could find a distributed neural representation that shifts the probability mass to the tokens "dog" and "give" instead of "yes" and "no", so this really is random causal structure that emerges from the massive number of parameters in the randomly initialized LLM. It's quite a neat result we plan to include in the paper! **Crucially**, we found that IIA drops to 0 for this run on all of our robustness checks from the paper. In addition, it drops to 0 if we use it on a different random LM. The results really nicely complement the ones we reported earlier for random rotation matrices on the highly structured, pretrained Alpaca model. These results will help calibrate people on the metric!
random baseline
Thank you, that finding clarifies things and also supports my point that the robustness check should be the *default* way of testing a circuit, rather than being considered a stress test. Would you be willing to slightly restructure in that way? I still would rather see more tasks, but the results from the random baseline satisfy me that at least the robustness test isn't completely broken as a validation method (although it appears that checking IIA without the distribution shift is somewhat broken, and outperforming a random baseline isn't completely satisfactory as a validation of the metric). Currently, while I would rather see more tasks rather than a case study on a single task, I'm willing to raise my score slightly reflecting the useful validation you've done. Don't be discouraged that it's still not a high score, as I think that the method is potentially very valuable.
We agree that these robustness tests are crucial evidence for our conclusions, and we will restructure to emphasize that our robustness checks and random baselines are crucial evidence for our conclusions and that they should be the default. Thank you for the meaningful critiques that lead to informative experiments!
Summary
This work proposes an extension to Distributed Alignment Search (DAS) called "Boundless DAS" that replace the brute-force search with learned parameters in order to scale the method to large language models (LLMs). The method evaluated on the Alpaca 7B model tasked to output "yes" or "no" to a simple numerical problem such as "is X between Y and Z?". Experimental analysis showed that the method is able to scale and find an alignment with two potential causal mechanisms. Further experiments also show that the method is robust to slight modifications in the prompt of the model (additional sentence in the input, change of output format).
Strengths
This work proposes a new explainability method that is scalable to LLMs, which is an important research question as the field quickly progresses towards larger models that become part of real-life products. Hence, research to understand the inner mechanisms of LLMs is very important. The proposed method is well-tested on various input/output perturbations of the target task, showing the robustness of Boundless DAS.
Weaknesses
- the technical aspect of the paper is challenging to understand for someone that is not familiar with the field of causality - the scope of the task used to evaluate the method is very narrow. Some discussion on how this can be applied to more critical scenarios where interpretability is required (such as the medical, financial, or legal domains) could improve the paper. - It is mentioned that a good causal model could also explain the errors of the system. However, the current method does not explain why the model fails in certain examples.
Questions
- Out of four possible causal mechanisms, two seem more plausible, but it is not clear which one is the one used by the model: "left boundary" or "left and right boundary"? Can we identify which causal method is the most likely being used, or is it the case that the model can use one or the other depending on the input?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
Some limitations are mentioned. However, one limitation of this work that should be mentioned is the relatively narrow scope of the target task being evaluated. This may be ok if the authors discuss how to transfer their method to more "realistic" scenarios in which interpretability is required (such as the medical, legal, or financial fields). Another possibility is that it is challenging or impractical to apply the same type of analysis in more complex scenarios. Either way, the paper could benefit from this type of discussion.
Summary
This paper introduces a method called Boundless DAS, an algorithm that can automatically test the "alignment" between human-specified symbolic algorithms and the computational structure found in the weights of a neural network. The authors apply Boundless DAS to a real-world 7B parameter language model called Alpaca and find evidence of the implementation of a particular symbolic algorithm that computes whether some number $q$ is in the range $[a, b]$.
Strengths
Really cool paper! * I like the idea of using gradient descent to create a correspondence between a symbolic algorithm and a set of neural network features. Continuous optimization feels a little easier to tackle (initially) than discrete combinatorial stuff. * It's impressive that this works on a real-world, billion-parameter model. * While the tricks to go from DAS to Boundless DAS seem reasonable and slick, my favorite part is the empirical section — inspecting the alignments that the algorithm discovers, looking for generalization, etc. Well done.
Weaknesses
Not much here. If I had to say anything, the paper is quite dense and hard to follow. For newcomers, it requires somewhat extensive background reading on causal abstraction and the previous DAS work(s). Otherwise, the writing is fairly clear, ideas are well-motivated, claims are well-supported and analyzed, etc.
Questions
* What kinds of tasks have you tried DAS or Boundless DAS on? Do you ever find gradient descent struggling to find a faithful solution? Do you have intuitions for what kinds of tasks this works well on, and what it doesn't? How about algorithms with lots of input variables, or complex ones requiring lots of intermediate ones? * I'm trying to get a sense for how far we can push Boundless DAS; what do you think? For any challenges you foresee, what do you think the right next steps are?
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
4 excellent
Contribution
4 excellent
Limitations
n/a
Summary
This paper proposes a method for interpreting neural models by learning an alignment of their representations to a specified interpretable causal model. The method extends prior work, namely Distributed Alignment Search (DAS), by addressing a key limitation that prevents the scaling of DAS. The limitation is overcome by learning a set of masks that selects subsets of neural representations to align to different parts of the interpretable causal model. The method is evaluated experimentally by aligning Alpaca representations to a set of correct and incorrect hand-written causal models.
Strengths
* The problem of interpreting large models is important. * The approach is reasonable and scales a previous approach. * The experimental results were convincing.
Weaknesses
* Section 3 was hard to read. See the questions below. * Section 3 did not feel self-contained. The source of intractability in DAS did not jump out to me. * Giving the computational complexity for DAS and boundless DAS would make the efficiency argument stronger, unless they are the same.
Questions
1. What is the value of introducing variable $Z$? It looks like only $X$ are used after section 3.1 2. Is $\Pi$ on line 99 used anywhere else? 3. In line 115, '$b_j$ is restricted to be a multiple of $b_{j-1}$ with a factor of 2'. However, $b_0 = 0$. What does this mean? 4. Why are target variables $Z_j$ bold but high-level variables $X_j$ not? 5. In Eq. (1), what is the difference between $F_N^*$ and $F_N$? Is $F_N$ defined? 6. What is the problem that was brute-forced in DAS, that $M$ learns to solve? Section 3.1 would be the best place to highlight intractability. 7. Line 37 would be another nice place to highlight limitations in scaling DAS. What is the largest dimensional representation it has been applied to? 1. What is the runtime improvement due to the method? 8. Line 216: "expectation" Clarifying the shortcoming of DAS, section 3, and giving the computational complexity would convince me.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
1 poor
Contribution
3 good
Limitations
The limitations were adequately addressed.
response to authors
Thank you very much for your clarifications, the contributions are now more clear to me. I would still like the authors to discuss in their revised paper the limiting factor for this method, which is (if I understand correctly according to "_The main bottleneck for us to try other tasks with Alpaca is that it is hard to find reasoning tasks that Alpaca does well._") that in order to find a causal alignment, the model must first be very good at solving the task on its own. I think this is very important to highlight for transparency reasons in the revised paper. The revised paper will benefit from some discussion around this, like why is this the case, and what future work could help mitigate this limitation. Thank you for your time and effort in this important research direction.
Thank you for your comments. We will revise our paper.
Yes, we will add discussion of this – thanks! It's really primarily an issue in the phase where we are seeking to motivate Boundless DAS. For that, we need to try to be sure that there is a systematic solution to identify. If we accept that Boundless DAS is a useful tool, then we can apply it to an under-performing model. It could be used to show that the model conforms to a very strange and problematic high-level causal model, or to accumulate some indirect evidence that the model doesn't implement any reasonable causal models. But one does need to trust the method at this stage, whereas we are currently seeking to build an argument for the method itself.
stress tests
Thank you, you've strengthened your results by checking a wider range of context. I think the method is potentially very valuable, and would eventually like to see this work published when it is ready. However, IIA itself doesn't seem to have any calibration for the user and I have no idea what a low value would entail. I believe that to evaluate your method, you need to compare the IIA to settings where you know that it should be low. To really be convinced that the value computed with your evaluation indicates a legitimate aligned circuit, I would want to see: - IIA after searching a random model (rather than comparing against the correct answer, you can compare against the answer that the model gives without intervention, so it's not true that it would be uninformative) - IIA for a known incorrect circuit of similar complexity to the ones found. - IIA on circuits for a task that is not consistently solved, in order to check whether intervening on the circuit actually corrects the model's output. Can you correct the answer by intervening on this circuit for the small number of cases where alpaca is otherwise incorrect? If not, it seems that this circuit is inadequate to explain the model's behavior even on the constrained task given. > A7: If a neural model performs at random with respect to a given high-level model which solves the task, we can know a priori that there is not a causal abstraction relation between the neural model and the high-level model. The problem is that I want to know the actual evaluation, not your a priori judgment of what the real circuit is. It is very easy to trick yourself into thinking that there is a meaningful circuit when there is not, and I don't think that I can accept this paper without knowing, at a minimum, whether you could trick yourself into believing that model had an interpretable circuit when it was in fact random. > A8,9: The target variables ... are sets of variables and the ... are variables. .... represents the causal mechanism of causal variable N before the distributed intervention, whereas ... This really doesn't answer my question very well, sorry. I think that your notation needs some rewriting and also if one of these is supposed to be a set of variables and the other is supposed to be a set of sets of variables, you should change the notation generally to reflect the different types. Generally, you need to work on your notation and define it in more detail so that the math is more readable. > A10: The boundary index variable marks how many neurons the intervention will use at the end. We penalize larger indexes since we want to push the intervention site to be smaller. So you have already chosen how many neurons the intervention actually uses? > A11: We restrict ... to be "a multiple of ... with a factor of 2 for simplicity." This really doesn't clarify anything. My point of confusion was this exact phase that I already read in the paper. What makes it simpler? Can you actually define this variable? Is it learned or is it preselected? > A13: In our case study, it is a latent boolean value that is abstracted to check whether the input price is above the lower bracket or below the higher bracket. If you know the answer, how is it a latent value? Overall, I am not satisfied as to how to assess a value for IIA, or how reliable your method is. I have described several experiments that could be run to calibrate the metric discovered by your method, but as is you have no actual evaluation for the method itself, instead using it straightforwardly on a very limited setting without any confirmation of what any given score indicates. I like the method, and I like the fact that you are trying to solve the problem of finding circuits at large scales in this way, but I would want to see the metric it computes be better calibrated through more experiments. To be clear, these problems exist in normal alignment search. However, your addition of more learned parameters exacerbates the possibility of finding a coincidentally high performing setting and fooling yourself into believing that the coincidence is meaningful. This is why I would like to see you try things like searching for irrelevant circuits. As a last comment, in your rebuttal you refer to whether this generalizes outside of the case of alpaca. I'm actually much less concerned about whether you generalize other language models, which I would accept you can, than whether you generalize to other tasks and circuits. You are advocating for this method on the basis of what is essentially a case study of searching a single set of parameters for behavior on a single very constrained task.
a possible way of checking a wider range of tasks
You could use the original tasks from geiger et al but with in context learning. I'm not sure if this would work. I just know that I don't entirely trust this method without more experiments and some point of comparison for how easy it would be be misled with the IIA metric.
Evidence that IIA is Well Calibrated
We appreciate the critical assessment of IIA, but we believe that **there are already results in the paper where we have low IIA and show that the method is calibrated**. According to our randomized alignment baseline, 50% IIA is the floor for performance on the boundary tasks and 60% IIA is the floor for performance on the mid point and bracket identity tasks. IIA is an accuracy metric, and so a gradient of success should not be surprising, and we consider values below the ALPACA model performance of 85% to be a failure to identify the causal variable. Small differences in accuracy can hide crucial aspects of the task! In figure 4 "Left Boundary", we can see that many of the locations we attempt to identify a variable are considered there are less than ten locations that we successfully identified the causal variable. **Of the remaining values, many are at random chance and others show some signal, but are below the threshold of strong evidence**. There are also low IIA values for two algorithms In figure 4, "midpoint" and "bracket identity" we see no strong evidence of these variables being represented. **All of the IIA values are low enough to be considered failures to confirm the hypotheses and most are at random chance**. **The lower IIA values in our main results provide calibration for the user** and show that this method cannot simply optimize to achieve perfect IIA wherever there is a representation that impacts the output. When compute is available, we will run a baseline with a randomly initialized model to add another point of calibration. Compute is scarce for us right now, though.
Follow-up on detailed technical questions
Follow-up on detailed technical questions **Here are responses to individual questions in the new responses:** > **Q8,9 (New):** “The target variables ... are sets of variables and the ... are variables. .... represents the causal mechanism of causal variable N before the distributed intervention, whereas …” **A8,9:** We will update our notations and method section combining suggestions from all the the reviewers. > **Q10 (New):** So you have already chosen how many neurons the intervention actually uses? **A10:** We define the neuron population, not the final neuron number that the intervention uses. For instance, we were to run Boundless DAS over a representation with a dimension of 4096, the initial neurons for the intervention is 4096. The boundary learning process will shrink down from that as we always incentivize for smaller boundaries. > **Q11 (New):** This really doesn't clarify anything. My point of confusion was this exact phase that I already read in the paper. What makes it simpler? Can you actually define this variable? Is it learned or is it preselected? **A11:** We restrict $b_j$ to be "a multiple of $b_{j-1}$ with **a factor of 2 for simplicity** in our case study. This restriction basically tells the alignment to start with the same number of swapping neurons for both the left and the right boundary. **It could be any factor.** The number of $b_j$ variables corresponds to the **learned** number of aligning causal variables at the same location. Now, we will elaborate our boundary learning process a bit more. Boundary index variable marks how many neurons the intervention will use at the end. We penalize larger indexes since we want to push the intervention site to be smaller. By using the boundary index variable, Eqn. 3 will generate a high-pass filter ($Mask_{s}$) between [$b_{j-1}$, $b_{j}$] and the gating factor within the bounds will be approximately 1.0 and approximately 0.0 outside the bound. We then apply this filter to the source representations, and the inverse filter (1.0 - $Mask_{s}$) to base representations before adding them up to get our "soft-intervened" representations. We will revise this section to be clearer in the next revision. > **Q13 (New):** If you know the answer, how is it a latent value? **A13:** Sorry about the confusion. We use “latent” to refer to variables that are not in the input or output explicitly. For instance, the boolean to check whether the input number is lower/higher than a bound is not explicitly in the input.
with a factor of 2 for simplicity
Could you find some way to paraphrase this? I just don't have any idea what this phrase is supposed to mean and it's not explained in the paper. So you've explained it to me three times (including in the paper), but each time you have used the exact same phrase, which I still don't understand.
more clarifications on "a factor of 2 for simplicity"
In our experiments, **we align at most 2 high-level variables**, when we learn left and right boundary variables. This means we only need to set two boundary variables to **split up a representation into three parts**: - (1) the part aligned with the left boundary - (2) the part aligned with the right boundary - (3) the part not aligned with any causal variable. **For convenience, we assumed that the dimensionality of the first two parts was equal** (i.e. the boundaries between variables related by $b_2$ = 2*$b_1$). This could be relaxed but seemed reasonable for the experiment in which the two variables were simply the left and right bracket values. In general, we can learn any number of boundaries, and we could alter the relationship between boundaries to be any differentiable function or independent. You may refer to our code for a better understanding. You can find this in **L245** in `code/models/llama/modelings_alignable_llama.py` in our attached `.zip` file.
factor of 2
So what you mean here is that you are restricted to a *branching* factor of 2 in the circuit?
Response
Thank you for answering my questions and clarifying notational issues. Based on the authors responses, I am reasonably confident the authors will make the writing in Section 3 clearer and provide more adequate background on the DAS method in the updated version, though I strongly urge the authors to make these changes, as it seems I was not the only reviewer who raised these points. I think that the main claim of the rebuttal that Boundless DAS is the "only method in this class of causal explanation methods that can even be applied at the scale of Alpaca or above" is a bit strong; causal tracing/causal mediation analysis is another method that is likely applicable at this scale, so I would be careful making such a claim without empirical support (e.g., runtimes). Additionally, the authors make claims about scaling beyond the Alpaca model used in the paper here, but I think the paper could benefit from more theoretical or empirical justification (or maybe just clearer writing) about why this is the case, such as done in your response to reviewer iGjk. Regarding my first question and the authors' general response "the network may be abstracted by many other causal models as well. Different causal abstractions highlight different explanatory aspects of model behavior.", I feel the paper would benefit from some discussion of how one might interpret or rectify various different causal models that provide different interpretations of model behavior. Without some means of aggregating them and/or resolving conflicts, the practical utility of this form of explanation might be limited (am I understanding this correctly?) I will keep my score as I feel that paper will be of high impact and tackles a difficult and valuable research question (though I defer to other reviewers who are more confident in their assessment of the math).
Thank you for your comments. We will revise our paper.
Thank you for your recent comments. We really appreciate this input. We are eager to make the work as accessible as possible, and this will help a lot in that regard. We will update our methods section to clarify further by addressing the feedback from our reviewers. Additionally, the connection with causal mediation analysis or causal tracing is interesting to explore. They are both intervention-based analyses methods and so there may be ways that we can use Boundless DAS to reduce the burden of searching for intervention sites for that method as well. Moreover, we'll expand the discussion in our limitations section to emphasize the assumptions behind our methods. We will also suggest potential future works to enhance the reliability of our interpretation results, such as aggregating or consolidating multiple alignment candidates.
Could you also provide the full computational complexity of DAS? I believe my concerns are addressed in the rebuttal. As all the concerns were presentation-related, I can only increase my score to weak accept without seeing the next version.
Thanks for the careful and detailed replies! I appreciate the inclusion of further DAS background. I think this is an interesting & potentially-valuable direction that deserves to be investigated further and discussed in the community.
Thank you for the detailed explanation of the DAS. It is definitely valuable in enhancing comprehension of the proposed approach. Regarding your comment on the novelty, I'm struggling to identify the distinctive technical contribution of the work. It seems to be a straightforward extension of DAS, incorporating an already existing technique (neural PDS [58]). Additionally, as highlighted by fellow reviewers, this extension hasn't been thoroughly assessed across various tasks, with outcomes predominantly confined to Alpaca. Due to these reasons, I would like to keep my score unchanged.
Decision
Accept (poster)