Summary
The paper provides an experimental study on localization claims of causal tracing and editing methods such as ROME. Existing literature uses localization results to decide which layer to edit for knowledge editing in large language models. The paper shows that, surprisingly, localization and edit success are nearly uncorrelated. In fact, one can edit a large range of MLP layers to achieve editing success. Yet, localization suggests that only a few early-mid-layers store the factual information. The paper then proposes new tasks and datasets to reconcile this issue.
Strengths
This is a very solid piece of empirical study. I find the conclusion convincing, as it is well supported by the experimental evidence.
The conceptual contribution is clear and novel. In particular, the uncorrelatedness of localization and edit success, as far as I know, has not been noticed by the literature. In fact, this is a somewhat surprising observation given the line of prior work such as ROME.
The paper is well-written.
Weaknesses
I do not find major weakness of the paper. I have some minor comments and suggestions.
The Appendix contains additional experiments on zero ablation. Another technique one can try is mean ablation and resampling ablation, over a known datatset such as CounterFact. This may be more principled than zero ablation. See the [Causal Scrubbing](https://www.lesswrong.com/posts/JvZhhzycHu2Yd57RN/causal-scrubbing-a-method-for-rigorously-testing#4_Why_ablate_by_resampling_) work for some discussion. I believe this wouldn't make much difference here, but it's worth trying.
Figure 1: Is this produced by aggregating over the CounterFact datasets? Would it be possible to provide a fine-grained look into this? What are the facts that appear to be stored at later layers?
Line 145: Here, the noise level is chosen to be 0.094 universally. However, in ROME, this is chosen to be three times the observed standard deviation of embedding norms as sampled over a body of text. Note that this quantity depends on both the data and the specific (trained) model. Should we also follow the convention of ROME here, to make a fair comparison? In my experience, causal tracing is somewhat sensitive to the level of noise. (I doubt it would change the conclusion, but it is worth looking into.)
Line 353: It might be worth a quick footnote here that not all examples have their tracing effect concentrated at the last subject token index. See Figure 11 of the original ROME paper for some natural instances.
Figure 10 in Appendix: y-axis should be Tracing Effect rather than Denoising Effect.
You used representation zero-ing in Appendix but representation zeroing in the main paper
Questions
From a mechanistic interpretability point of view, could the author(s) provide more thoughts on the effectiveness of causal tracing (in locating which layer *mechanistically* stores the facts)? In particular, would you argue that the model computation is really not as concentrated as causal tracing suggests? Is it rather a diffuse and incremental process, with lots of redundancy? It seems from the window size = 1 experiment (Figure 10 in Appendix), the picture clearly shows that no single layer gets much tracing effect. Thus, would you agree that the sliding window patching is not principled and could yield a false illusion that the information is sharply concentrated?
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The author(s) addressed several limitations already. I believe it's adequate.