Author Response
### Thank you for acknowledging many positive aspects of our work; we can alleviate many of the weaknesses you mention with clarifications presented here and writing updates to our PDF. We've updated text in diagrams to be as large as possible given the page constraint, and made another pass to fix typos and put citations in brackets; thank you for these catches. Updated PDF coming soon!
> *"The claim in the lines 518-519…cannot be deduced from the analysis of the checkpoints of only one particular Olmo 0724 7B base model (however, at lines 79-81 you make much more humble version of this claim, with which I agree).”*
Thank you for pointing this out; we will update this sentence in our new PDF (coming shortly) to instead reflect the more humble version of the claim.
> *"In the lines 76-78…you only showed this effect for one non-standard letter set, and, again, I see only Olmo 7B experiments here (please, correct me if I'm wrong).”*
When we say “the model’s hidden state”, we are referring to the Olmo 7B Instruct model mentioned earlier in the sentence. We are explicit in the paper that this result is for the Olmo model (lines 409 and 416), and thus represents one means by which models produce OOD letters (line 41). We'll further clarify this in the updated PDF though, and you're right that it will be valuable for us to add experiments for Llama and Qwen.
As for other non-standard answer symbols, we did observe this effect consistently in our preliminary experiments, but did not include these results in the paper. This is valuable feedback and we will add results on another randomly-selected letter set to show that our findings are robust.
### Your questions:
> *"I don't understand the caption for the Figure 1, especially the sentence "Finally, when we switch to more unusual answer choice symbols, one behavior by which models adjust is to initially operate in the space of more standard tokens, such as A/B/C/D, before aligning probability to the correct symbols at a late layer".”*
This is referring to our finding in lines 76-78 and in Section 5 lines 407-419 under the heading “Some answer symbols are produced in a two-stage process”: that for the Olmo 7B Instruct model, hidden states initially assign high logit values to expected answer symbols (A/B/C/D) before switching to the symbols given in the prompt. We'll try to adjust the phrasing to make this clearer.
> *"Fig.9: Do you have such curve for any other model, aside of Olmo 0724 7B base?”*
No, and unfortunately we are unable to produce one. The Llama and Qwen developers did not release any intermediate model checkpoints, and the Olmo 1B model doesn’t do well enough on the synthetic task at the end of training to warrant constructing one (see Fig. 2).
> *"Fig.4: I don't understand the purpose of making this figure. What should it show to us?”*
The purpose of the figure is to show that trends in promoting answer choices in vocabulary space are largely similar across tasks, despite the fact that the difficulty and subject matter of the tasks vary substantially and model performance differs as a result (i.e., from left to right: 55%, 51%, and 100% accuracy). We elaborate on this in lines 392-394, but if additional clarification or discussion would be valuable, we are happy to add. We’ve also added the accuracies to the figure caption.
> *"Fig.8: Can you, please, elaborate the captions a), b), c)?”*
Plots b) and c): this is the difference in logits on the answer choice tokens produced by a hidden state projected to the vocabulary space (described in Section 4.2, specifically lines 254-255). This is equivalent to the “logit difference” lines in Figures 5a and 5b, except now at the more fine-grained level of individual attention heads instead of hidden state outputs. We plot this to show the extent to which attention heads specialize *by letter* (by comparing how the two plots differ). Additionally, while the model is ultimately producing positive values (blue) for the instances in both graphs that result in correct predictions, the plots show that not all attention heads contribute positively towards the model predicting the correct answers.
Plot a): this is the sum of logit values for all answer choices (logit of A + logit of B + logit of C + logit of D) using the same vocabulary projection method. This is equivalent to summing the A, B, C, and D lines in Figure 5a, except now at the more fine-grained level of individual attention heads instead of hidden state outputs. We plot this to show the overlap between plots a), b) and c): many attention heads appear to play a role in both generating *any* valid answer choice symbol and outputting the *correct* symbol (lines 478-479).
We will include additional context for these plots in the paper.