Response #2
Thank you for your helpful review.
> "The paper details the phenomena of overthinking and false induction heads, but it doesn’t fully articulate how these insights could be applied practically to improve model behavior. For instance, while head ablations are discussed as a method to reduce overthinking, the practical impact of these modifications on model functionality and broader applicability is not thoroughly explored in real-world scenarios i.e. how does one know when the input examples are likely to be false, and that heads ought to be ablated?"
While we were mainly focused on understanding how models process inaccurate prompts, we believe our results suggest some directions for mitigation. One way to identify if input examples are false might be to look for sharp changes in prediction during the model’s processing. Another possibility would be to train a probe on the model activations to determine if the input examples are false. Moreover, in our experiments we found that ablating the heads / decoding early led to large improvements given incorrect demonstrations, with no or much smaller effects given correct demonstrations. Therefore, at least in our ICL setting, early exiting improved performance without needing to determine which input examples are false.
> “The methodology involves decoding from intermediate layers and identifying false induction heads, but the paper could provide a more explicit explanation of these processes i.e. how predictions were extracted and analyzed at each layer, and how the heads contributing to overthinking were identified and analyzed.”
Thank you for this feedback. Here is a more detailed explanation, which we can add to the paper if you find it clear:
To extract predictions from each layer, we used the logit lens (Nostalgebraist, 2020): we got logits by applying the unembedding matrix to the post layernorm activations after each layer (equation on page 5).
We chose the 5 heads with the highest context following scores (equation on page 7). We selected the heads on our “Unnatural” synthetic dataset to demonstrate that these heads generalize to more realistic datasets.
Please let us know if there is anything further we can clarify in this procedure.
> “The paper could better acknowledge that the 'incorrectness' of permuted labels can vary based on the dataset and what each label signifies. Some permuted labels might be clearly wrong, while others could be more ambiguous, and providing specific examples for each dataset could clarify this aspect and help to understand the results.”
Our understanding of what you mean by ambiguous is that in some datasets, some labels have similar meanings. For example, in DBPedia, the class labels “Plant” and “Nature” are similar, so assigning the label “Nature” to a plant would not be clearly wrong.
Assuming this is what you had in mind, this ambiguity generally should not affect our conclusions, since we use a new permutation for each input sequence and so most permutations send “Plant” to a different, unrelated label. Therefore, this effect gets averaged out.