We believe the key parts of the review were addressed in the main rebuttal, but we include this comment for completeness.
> **Q1:** How are the token frequencies computed for the second half of the paper?
The empirical token frequency distribution that we compute is the unigram distribution: the distribution of tokens (i.e., entries in the vocabulary) over the whole training corpus. More specifically, the frequency of token $t \in \mathcal{V}$ over corpus $\mathcal{C}$ is computed as \# of occurrences of $t$ in $\mathcal{C}$ / $|\mathcal{C}|$. This distribution depends on the training data, and ideally, it should be computed for a specific model using the exact training corpus that the model was trained on. We could achieve this for the Pythia models as we had access to the data statistics of The Pile. For GPT-2, since the exact training data is not available, we used a randomly sampled 500M-token subset of the OpenWebText corpus as a proxy for the original distribution (as mentioned in Appendix G). Different training data distributions might lead to different token frequency distributions. However, we expect a model to align its output distribution with the token frequency distribution it was exposed to during training.
> **Q2:** How many entropy neurons usually exist?
The extent to which a neuron is considered entropy-regulating depends on the fraction of its effect that is mediated by the LayerNorm. This quantity varies on a continuous scale, and defining a specific number of entropy neurons would require setting a hard threshold on this measure, which is somewhat arbitrary and difficult to determine precisely. However, in our analyses, we observed that the number of neurons for which the LayerNorm-mediated effect is substantial typically accounts for around 0.1-0.3% of the MLP dimensionality.
> **Q6:** In Figure 4, what is the reciprocal rank exactly? Is a higher value better?
The reciprocal rank is computed as $ \frac{1}{r}$, where $r \in \{1, \dots, |\mathcal{V}| \}$ is the position of the correct next token in the list of all vocabulary tokens sorted by their predicted probability mass in descending order. For example, if the correct next token is ranked 1st in the predicted probability distribution $P_{\text{model}}$ (i.e., the next token prediction is correct), the reciprocal rank is 1. Lower values indicate that the correct token is ranked lower in the predicted probability distribution (i.e., that the prediction is worse).
> It will be useful to state somewhere that your neuron indexing strategy is <layer>.<neuron>, which may be unclear to a new reader.
Thank you for pointing this out. We will add a sentence to clarify the notation.
---
[1] Wang, X., et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.
[2] Hou, B.,et al., Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling. ICML 2024.
[3] Jiang, Z., et al.,. How can we know what language models know?. TACL
[4] Kadavath, S., et al., Language models (mostly) know what they know. arXiv.
[5] Lin, S., et al., Teaching models to express their uncertainty in words. arXiv 2022.
[6] Si, C., et al., Prompting GPT-3 To Be Reliable. ICLR 2023.
[7] Tian, K., et al., Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. EMNLP 2023.
[8] Burns, C., et al,. Discovering latent knowledge in language models without supervision. ICLR 2023.
[9] Azaria, A. and Mitchell, T., The internal state of an LLM knows when it's lying. arXiv 2023.