Confidence Regulation Neurons in Language Models

Despite their widespread use, the mechanisms by which large language models (LLMs) represent and regulate uncertainty in next-token predictions remain largely unexplored. This study investigates two critical components believed to influence this uncertainty: the recently discovered entropy neurons and a new set of components that we term token frequency neurons. Entropy neurons are characterized by an unusually high weight norm and influence the final layer normalization (LayerNorm) scale to effectively scale down the logits. Our work shows that entropy neurons operate by writing onto an unembedding null space, allowing them to impact the residual stream norm with minimal direct effect on the logits themselves. We observe the presence of entropy neurons across a range of models, up to 7 billion parameters. On the other hand, token frequency neurons, which we discover and describe here for the first time, boost or suppress each token's logit proportionally to its log frequency, thereby shifting the output distribution towards or away from the unigram distribution. Finally, we present a detailed case study where entropy neurons actively manage confidence in the setting of induction, i.e. detecting and continuing repeated subsequences.

Paper

Similar papers

Peer review

Reviewer S9nW9/10 · confidence 5/52024-07-03

Summary

This paper studies how large language models (LLMs) regulate uncertainty in next-token predictions through specific components: entropy neurons and token frequency neurons. Entropy neurons, identified by their high weight norm and minimal direct impact on logits, influence model confidence by operating within an unembedding null space. Token frequency neurons adjust logits based on token frequency, modulating the output distribution toward or away from the unigram distribution. The study includes a detailed examination of these neurons in various models, demonstrating their role in managing prediction confidence, particularly in repeated sequence scenarios.

Strengths

- The extensive experimental validation, including ablation studies and cross-model analysis, gives empirical support to the theoretical claims, enhancing the overall robustness and reliability of the study. - The paper is very well written, and it was easy to follow through even though the topic is rather complex. - I think that the paper provides an good mechanistic explanation of how entropy neurons operate through the unembedding null space and LayerNorm, improving our understanding of their indirect impact on model predictions. Overall, I find the paper's findings very valuable, the methodology is rouboust, and the authors adressed themselves the few limitations of the paper. I honestly enojoyed reviewing this paper.

Weaknesses

- As the authors say in the "limitations" section, they use entropy and distance from the token frequency distribution as proxies for confidence, which may not fully capture the complexity of confidence in language models. - The extent to which entropy neurons influence model output via LayerNorm seems to change across models. This variability suggests that model architecture and training parameters could play a important role.

Questions

Please, read my questions and suggestion as a starting point for a follow-up paper. - Could you provide a more precise definition of confidence in the context of LLMs? - While LayerNorm plays a crucial role in the functioning of entropy neurons, what might be the impact of other normalization techniques (for example, BatchNorm or RMSNorm) on these neurons? - I'd suggest also to perform an analysis of how specific features of the models and training hyperparameters influence the effectiveness of entropy and token frequency neurons.

Rating

9

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors properly addressed all the possible limitations of the paper.

Reviewer Thsu5/10 · confidence 4/52024-07-10

Summary

The paper investigates specific neurons in LLMs (termed "confidence regulation neurons") that modulate the uncertainty of the next token prediction by modulating the output distribution. First, entropy neurons modulate the overall entropy of the output distribution by writing to an effective null space of the unembedding matrix, thereby influencing the residual stream norm with a minimal direct effect on the logits themselves. Second, token frequency neurons modulate the predictive distribution proportionally to the token frequency in the training data.

Strengths

- The paper clearly written and well-organized, making complex concepts accessible. - It provides deeper insight into the role of entropy neurons in regulating the confidence of LLMs through the unembedding null space. - It introduces token frequency neurons, a type of neurons that have not been discussed in prior work. - It demonstrates the presence of these entropy and token frequency neurons across various models.

Weaknesses

- **Novelty**: The authors claim that they provide a "novel example of how language models can use LayerNorm to indirectly manipulate the logit values" and that prior studies "fail to account for the normalization effects imposed by LayerNorm" (lines 160 to 163). However, the mechanisms of entropy neurons have already been discovered in previous work [1]. For instance, Gurnee et al. (2023) show that entropy neurons "modulate the model’s uncertainty over the next token by using the layer norm to squeeze the logit distribution, in a manner quite similar to manually increasing the temperature when performing inference.". Although prior studies do not differentiate between the total and direct effect of an entropy neuron on the output distribution, the novelty of the analysis is not clear-cut. - **Clarity**: Individual neurons are referred to as *layer.position* and simply *position* interchangebly (e.g. 11.2378 and 2378). Since only neurons in the final layer were investigated, referring to the layer is redundant. Also, the authors should consider further simplifying the name of the neurons, as the exact *position* does not add immediate value to the reader. - **Interpretability**: The interpretation of the results is sometimes unclear. For instance, the authors observe that token frequency neurons suppress common tokens and promote rare ones. They suggest this indicates the model is biased to predict common tokens more frequently than their actual occurrence. However, an alternative interpretation could be considered. The model might also assign high probability to a single rare token while assigning almost no probability to other rare tokens. Promoting all other rare tokens to match their token frequency would bring the output distribution closer to the token frequency distribution, while increasing entropy and lowering confidence. In other words, the model's bias might not be just towards common tokens but also towards certain rare tokens. Such alternative interpretations would also resolve the counterintuitive explanation. The authors should provide more insights to support their interpretations. --- [1] Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. Universal neurons in gpt2 language models.

Questions

- Why did the authors analyze 6 entropy neurons but only 5 token frequency neurons? - In Figure 5(a), are entropy, loss, and neuron activations really all on the same scale (single y-axis)? - Why does the induction case study not consider the novel token frequency neurons but only the known entropy neurons?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors adequately addressed limitations.

Authorsrebuttal2024-08-12

Before this phase of the discussion period ends, we wanted to check in with the reviewer on whether we have addressed your concerns with our work?

Reviewer Thsu2024-08-13

Thank you for the rebuttal; it addressed the majority of my concerns and questions. My current score reflects my overall assessment of this work pretty well, so I have decided to maintain the score.

Reviewer BDF87/10 · confidence 3/52024-07-11

Summary

This paper investigates two kinds of neurons by which transformer language models calibrate their predictions. These are (1) “entropy neurons”, which can affect logit values, but do not promote specific tokens, and (2) “token frequency neurons”, which influence a model’s likelihood of outputting bigram word statistics.

Strengths

- Calibration of model confidence is an important area of study for trustworthy deployment of ML systems. - The experiments attempt to show that both kinds of neurons for model calibration arise across various model sizes and architectures up to LlaMa-2-7B, suggesting this description is general to an extent (varying success across models). - The results have nice synergy with existing work, providing a clearer understanding of how these previously discovered neurons can influence model behavior.

Weaknesses

- It is unclear what the connection is between “token frequency neurons” and “entropy neurons”. Do these two kinds of neurons interact (if so, how?) or are they separate mechanisms by which models calibrate logits? As this is a major part of the paper, it should be made clear. - The case studies provided are pretty simple settings. Studying the effect of calibration neurons on more real-world tasks like question answering - where there is a small set of potential answers and seeing if and how the same mechanisms apply would help strengthen the generality of the claims made. (This is noted by the authors in the limitations section, and I agree with them).

Questions

- The total effect is much larger than the direct effect for both sets of neurons. For the “token frequency neurons”, does this mean they are influencing other components that actually promote bigram statistics? Or what exactly is their effect mediated by? It is unclear to me how the same direction could promote the most common bigrams for every distinct token in the vocabulary. - Are entropy neurons and token frequency neurons only studied at the last layer of the model? Do they appear elsewhere, and are just strongest at the last layer? - Does the discovery of these neurons influence how we should think about using direct logic attribution as a way to understand components and their interactions? Lines 70-71 briefly mention this, but could you can briefly expand on what is meant here?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors do point out a number of limitations to their study, which are valid and could strengthen the paper when/if addressed. Though many may not be in scope.

Reviewer C4Wm7/10 · confidence 3/52024-07-13

Summary

This work studies a small but potentially important subset of neurons in the final layer for a trained transformer model that appear to regulate the confidence of a model (proxied by the variation in entropy of the model's output). Specifically, the paper builds upon entropy neurons found in previous work, extending the analysis further to show that these entropy neurons (which have high norm but low composition with the unembedding matrix) essential modify the null space of the unembedding matrix. This effectively scales the logits without changing the relative ranks of the tokens, thus regulating confidence. The paper continues this exploration by finding "token frequency" neurons, which are neurons that move the model's output distribution towards unigram token frequency distribution. The work claims that this is another form of confidence regulation, essentially "moving" the distribution towards unigram frequencies when the model is not confident about its prediction. The paper provides the methodology for finding these neurons, as well as performs ablation/intervention experiments on these neurons to measure the difference in "confidence". Experiments are conducted on several pre-trained models, including GPT2, Pythia, LLaMA-2, Gemma and Phi-2. The paper also ends with a specific analysis of "induction"-behavior depicted in recent mechanistic interpretability works, studying how entropy/token-frequency neurons affect the output distribution when sequences are repeated in the output.

Strengths

- The overall paper is well written, describing its motivation well, as well as the methodology. The figures presented do a good job at compressing the useful information in an accessible form. - The experiments conducted as convincing, and improve upon the findings of the previous works on which this paper is built upon. The experiments are also conducted on several models, which further showcases the universality of the findings. - The paper focuses on an important problem of understanding transformer models (although the immediate practical applicability of the findings is limited, it still answers important questions)

Weaknesses

- While the results presented in the paper are indeed exciting, they raise a lot of questions (see Questions section) that are easy to answer but have been ignored by the paper. For instance, while the paper has run experiments on many models, there is very little discussion of these results. The paper would be much stronger if some additional discussion revolving around model and vocabulary sizes was included. - The paper mentions a few existing works on confidence regulation and calibration, but does not tie the work done in this paper within the context of this broader field.

Questions

- How are the token frequencies computed for the second half of the paper? Are these sub-word frequencies? On what dataset are these computed, and do you expect the training data distribution to have any effect on the results? - How many entropy neurons usually exist? From the plots, it seems this number may be somewhere in the single digit range? Explicitly stating this will still be useful information for the reader - Can you hypothesize why Gemma 2B has few to no entropy neurons? - Relatedly, what is the effect of model size on the number of these entropy/token-frequency neurons? Should one assume that model size has no effect? What about the vocabulary size? - Token freq is confusing; by changing distribution differently for every token, aren’t we changing the logits necessarily? - In Figure 4, what is the reciprocal rank exactly? Is a higher value better? - One thing that is unclear is how token-frequency neurons are regulating confidence specifically? I understand that they are modulating the output distribution towards unigram frequencies, but this affects both the logit ranks as well as the variance, so the effect on "confidence" specifically is unclear. I would appreciate some clarity on this. ### General comments - It will be useful to state somewhere that your neuron indexing strategy is `<layer>.<neuron>`, which may be unclear to a new reader.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Authors have addressed limitations adequately.

Authorsrebuttal2024-08-07

We believe the key parts of the review were addressed in the main rebuttal, but we include this comment for completeness. > **Q1:** How are the token frequencies computed for the second half of the paper?  The empirical token frequency distribution that we compute is the unigram distribution: the distribution of tokens (i.e., entries in the vocabulary) over the whole training corpus. More specifically, the frequency of token $t \in \mathcal{V}$ over corpus $\mathcal{C}$ is computed as  \# of occurrences of $t$ in $\mathcal{C}$ / $|\mathcal{C}|$. This distribution depends on the training data, and ideally, it should be computed for a specific model using the exact training corpus that the model was trained on. We could achieve this for the Pythia models as we had access to the data statistics of The Pile. For GPT-2, since the exact training data is not available, we used a randomly sampled 500M-token subset of the OpenWebText corpus as a proxy for the original distribution (as mentioned in Appendix G). Different training data distributions might lead to different token frequency distributions. However, we expect a model to align its output distribution with the token frequency distribution it was exposed to during training. > **Q2:** How many entropy neurons usually exist? The extent to which a neuron is considered entropy-regulating depends on the fraction of its effect that is mediated by the LayerNorm. This quantity varies on a continuous scale, and defining a specific number of entropy neurons would require setting a hard threshold on this measure, which is somewhat arbitrary and difficult to determine precisely. However, in our analyses, we observed that the number of neurons for which the LayerNorm-mediated effect is substantial typically accounts for around 0.1-0.3% of the MLP dimensionality. > **Q6:** In Figure 4, what is the reciprocal rank exactly? Is a higher value better? The reciprocal rank is computed as $ \frac{1}{r}$, where $r \in \{1, \dots, |\mathcal{V}| \}$ is the position of the correct next token in the list of all vocabulary tokens sorted by their predicted probability mass in descending order. For example, if the correct next token is ranked 1st in the predicted probability distribution $P_{\text{model}}$ (i.e., the next token prediction is correct), the reciprocal rank is 1. Lower values indicate that the correct token is ranked lower in the predicted probability distribution (i.e., that the prediction is worse). > It will be useful to state somewhere that your neuron indexing strategy is <layer>.<neuron>, which may be unclear to a new reader. Thank you for pointing this out. We will add a sentence to clarify the notation. --- [1] Wang, X., et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023. [2] Hou, B.,et al., Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling. ICML 2024. [3] Jiang, Z., et al.,. How can we know what language models know?. TACL  [4] Kadavath, S., et al.,  Language models (mostly) know what they know. arXiv. [5] Lin, S., et al., Teaching models to express their uncertainty in words. arXiv 2022. [6] Si, C., et al., Prompting GPT-3 To Be Reliable. ICLR 2023. [7] Tian, K., et al., Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. EMNLP 2023. [8] Burns, C., et al,. Discovering latent knowledge in language models without supervision. ICLR 2023. [9] Azaria, A. and Mitchell, T., The internal state of an LLM knows when it's lying. arXiv 2023.

Reviewer C4Wm2024-08-13

Thank you for your response, some of my queries have been answered. I have gone through the other reviewers' comments and will maintain my score.

Reviewer BDF82024-08-08

Thank you for the reply

Thank you for the detailed reply. After reading the authors' rebuttals and other reviews, all of my questions/concerns have now been addressed and I think the paper would be a solid contribution to the conference. As such, I have raised my score to a 7.

Reviewer S9nW2024-08-10

Thank you for your detailed responses, I appreciate the clarifications provided, particularly regarding the definition of confidence in LLMs and the impact of different normalization techniques on entropy neurons.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC