Massive Activations in Large Language Models

Summary

This paper identifies a new phenomenon called “massive activations” in Large Language Models (LLMs), which shows that the hidden state vectors in intermediate layers at certain token locations have a few exceedingly large coordinates. The finding extends the attention sink phenomenon of Xiao et al. (2023) in the sense that these tokens are precisely the tokens that attract attention scores. The paper reports several qualitative properties such as their appearance and their difference in different LLMs. The paper performs intervention studies and further some small-scale pretraining experiments to demonstrate that massive tokens act like a fixed bias in the attention mechanism.

Rating

8

Confidence

4

Ethics flag

1

Reasons to accept

Overall I think the paper reports a very intriguing and potentially important phenomenon in LLMs. The phenomenon could be of immediate interest to the theoretical understanding / mechanistic interpretability communities. The experiments are also quite comprehensive and provide a useful reference point for practitioners. * The phenomenon extends the attention sink phenomenon and in my opinion actually gives (at least a partial) explanation of the attention sink phenomenon. * The finding that massive tokens also appear in *delimiter tokens* (Sec 2.2) is interesting and (kind of) new even to the attention sink literature. Previously people only showed that these “information-less” tokens act as attention sink in encoder LMs or ViTs. This paper shows that delimiter tokens can also attract attention in standard autoregressive decoder LMs. * The intuition that attention sink acts as fixed attention biases is very sensible and the paper provides some justifications. * The study is quite comprehensive in both the models (LlaMa 7B & 13B, Mistral 7B, Phi-2) as well as the various qualitative properties. * The presentation and in particular the visualizations are quite well done.

Reasons to reject

* The pretraining experiment (Sec 4.3) with the k’v’ variation as in Eq (3) is not fully satisfactory in showing that massive tokens act as a fixed bias. - Eq (3) does not really differ from the “GPT2 with sink token” baseline, as it seems mathematically almost equivalent (maybe up to some minor reparametrization, since Eq(3) did not parametrize a hidden state vector but rather one k’v’ vector for each head). So even though k’v’ method removes massive activation in other tokens as reported, I am guessing this special k’v’ token still attracts (say >90%) attention, so this method still preserves attention concentration. - (An interesting test would be whether k’v’ “correspond” to massive activations or their post-layernorm versions, but here it does not sound possible since you only parametrized k’v’, not the pre-KV projection versions.) - On the other hand, Eq (8) in appendix seems like a more ideal version of fixed attention bias, and addresses the unbalancedness in the softmax as well as the massive activation phenomenon. Unfortunately pretraining with it only works to some extent (it reduces the top activation magnitude from 2K to 600, but the ideal target is like 100-200). Again, I am wondering if the attention scores are still concentrated here (and I am guessing they are)? - If Eq (8) cannot be made to work to fully remove massive activation, it then creates the question of whether attention concentration is fundamental (for achieving good LM performance). Have the authors thought about other ways of removing massive activations / attention concentration? * No immediate practical applications, but I don’t think that’s a major concern given the potential importance of the phenomenon. I also wonder if the authors have thought about alleviating training instabilities by mitigating massive activation and/or attention concentration (kind of related to the above question)?

Questions to authors

See the “weakness” part.

Summary

This paper introduces an outrageously large value sometimes appears in a hidden vector in a large language model. This paper calls such a large value "massive activation". The authors conducted several analyses on the massive activations. Then, the authors show situation which the massive activation appears in, and the relation between attention calculation and the massive activation.

Rating

4

Confidence

4

Ethics flag

1

Reasons to accept

The massive activations, that are introduced by this paper, are interesting phenomena. The analyses on massive activations might have an influence on future studies on LLM behaviors and training dynamics of LLMs.

Reasons to reject

This paper introduces an interesting phenomenon, the massive activation, but I think this is an ongoing study because this paper does not reveal the reason of the massive activation and the effect on LLM behaviors. For example, the relation on the stabilization of LLMs. Section 4 of this paper implies that the massive activations have a role such as a bias term. PaLM paper (https://arxiv.org/abs/2204.02311) clams that it important to remove bias terms to stabilize the pre-training of LLMs. These suggest that massive activations might have a serious influence on the stabilization of LLMs. Like this example on the stabilization, the authors should conduct more investigations to reveal why massive activations appear in.

Summary

This analysis paper studies the phenomenon of massive activations, which is defined in this paper as activations with magnitude larger than 100 and 1,000 times larger than the median magnitude of its hidden state. The paper had the following main findings: - Massive activations widely exists in different models, largely in the middle layers, and occurs on fixed dimensions of a model and often on frequent tokens - Setting massive activation to 0 significantly changes model behaviors. - Massive activation causes attention to be concentrated, and act as bias terms in attention modules. Training models with a bias term in the attention module can eliminate massive activations. In summary, I think the general presentation quality is very high, forming a very well-motivated and coherent story that's mostly easy to follow. I do have some doubts about the impact of this work, because I fail to see how such understanding can help us make improvements to the current transformer architecture. I'm hoping the authors to shed more lights on those doubts.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

- This paper presents a novel, interesting, and thorough analysis that advances the communities understanding of massive activation phenomenon in the transformer architecture. - The presentation quality is very high. The clarity of the experimental setting is good. The transitions in between different parts of the analysis are well motivated. The writing is mostly easy to follow.

Reasons to reject

- Although the authors provided great insight into reasons why massive activations occur in transformers, I have some doubts on what steps the community can take on those findings. Some discussions around that theme would be very helpful -- for example, should people start to add a bias term on the attention modules? Should people be more careful with normalization in the attention module? What improvements should people expect from them? Is it better performance or just better training stability? Answering these questions can potentially help amplify the impact of the paper. - Analysis stopped at 13B models and decoder-only models. I would be curious to see if the phenomenon generalizes to encoder-decoder models, or even larger models, e.g. 175B BLOOM models.

Questions to authors

- For "setting massive activation to a certain value" experiment in Section 3, did you try to set it to a non-zero regular magnitude value (e.g. mean of the hidden state)? I wonder if 0 has some kind mathematical specificity here that might have caused the significant change. - For Section 4.2, I'm not quite sure if I'm following the transition between layer-norm to output decomposition -- they feel somewhat unrelated to each other. Can you explain how one affect another? Also, have you tried to run experiments with the attention module in the original transformer paper? I believe there is no layer-norm before the linear transformation.

Summary

This paper identifies the phenomenon of massive activations in large language models (LLMs), where very few activations exhibit significantly larger values than others. It extensively studies the layers where massive activations reside, their importance to LLMs' capabilities, and their role as bias in the self-attention mechanism. Finally, the analysis is further extended to ViTs.

Rating

7

Confidence

5

Ethics flag

1

Reasons to accept

1. The identified existence of massive activations is interesting. Although related work like FLAP [1] has provided similar observations, a comprehensive study of this phenomenon could provide useful insights for the community. [1] "Fluctuation-based Adaptive Structured Pruning for Large Language Models", Y. An et al., AAAI'24. 2. The performed analysis regarding the importance and role of massive activations is technically sound and extensive. 3. The writing is logically clear and coherent.

Reasons to reject

1. Some of the observations lack deeper analysis. For example, massive activations seem to serve as an important bias in the attention mechanism, while they mainly focus on semantically unimportant tokens like delimiters. The authors are expected to provide a clearer explanation regarding this gap. 2. It is not clear whether massive activations are desirable or whether they will disappear with larger-scale pretraining. The authors are expected to evaluate larger LLMs pretrained on larger corpora to determine whether the percentage of massive activations will increase with the model scale and data scale.

Response to author feedback

Thank the authors for their response, which properly addresses my concerns. I will keep my original rating.

Thanks for the reponse

The authors have addressed all my concerns. I plan to maintain my original score. Please consider adding the practical implication section of the response to the paper, maybe replacing the final sentence of the paper -- it's a more meaningful way to end the paper rather than saying "we observed this, hopefully it helps" :)