Transformers need glasses! Information over-squashing in language tasks

Summary

This paper presents an in-depth analysis of decoder-only Transformers, focusing on their limitations in handling information propagation. The authors identify two key phenomena: "representational collapse" due to "over-squashing" (line in Graph Neural Networks). These issues lead to a significant loss of information, especially in tasks involving counting and copying, which require precise handling of individual tokens. The paper combines theoretical analysis with empirical evidence to demonstrate these problems.

Strengths

1. The paper provides a theoretical framework to understand the limitations of decoder-only Transformers. The concepts of representational collapse and over-squashing are well-formulated and offer valuable insights into why Transformers struggle with certain tasks. 2. Very strong theoretical analysis is provided in paper, but the clarity of narrative is preserved. The quality of presentation is excellent. 3. The authors provide empirical evidence from contemporary LLM, specifically Gemini 1.5 and Gemma 7B, supporting their theory. They also provide the analysis of effect of floating point precision. 4. Authors carefully provide all the necessary details about experiment.

Weaknesses

1. The experiments are primarily focused on specific artificial tasks (counting and copying), but the study lacks the analysis of the real-world texts. It would be beneficial to include some statistics of difference of tokens' representations on the standard text corpora. 2. Authors provide only one artificial solution to split the sequence by different token. 3. The theoretical analysis makes some simplifying assumptions, such as treating attention weights as independent of the input. While the authors justify these assumptions, it would be beneficial to explore their impact on the results more thoroughly.

Questions

1. Could you please include some statistics of difference of tokens' representations on the standard text corpora? It would be interesting to see does representational collapse happen in natural texts. 2. How can you deal with representational collapse in real-world scenarios? Wouldn't it be worse to include random token for the information? 3. Could this theoretical result be generalized to non-causal language modeling (different attention mask)? Also here are some small remarks: Line 331 typo: summariesd Line 339 space in "phenomena : representational"

Rating

7

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

1. Theoretical analysis relies on several simplifying assumptions, which are justified but may not fully capture the complexities of real-world models. 2. Empirical validation is conducted only on two specific models

Summary

The paper provides a theoretical and empirical analysis of the final representations of transformers, revealing the phenomenon of representational collapse.

Strengths

- The paper is well-written. - It highlights a problem in transformers that causes them to fail on a large set of tasks (assuming this extends to addition, etc.). - The paper provides empirical evidence supporting the theoretical claims, demonstrating the real-world relevance of the identified issues. - The impact of low-precision floating-point formats is interesting, which is highly relevant to current everyday practices

Weaknesses

- There is no strong solution to this problem. Although the authors provide a simple approach, it is challenging to understand how to apply this approach in a practical setting. - The problems seem to be tied to the positional embeddings, as the authors stated. NoPE (https://arxiv.org/abs/2305.19466) and no causal mask would make this problem impossible. However, there are no experiments on how certain positional embeddings might be better than others.

Questions

- What do the authors believe is the breadth of tasks affected by this fundamental issue in transformers? - What are the positional embeddings used for the experimental settings? Can these affect the representational collapse? - Minor: Some figures, like Figure 2, have a very small font size, making them hard to read.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

See Questions and Weaknesses.

Summary

In the paper, the authors first discuss a phenomenon occurring in LLMs that they call "representational collapse". They provide empirical evidence of the phenomenon in state-of-the-art language models and they provide a theoretical justification for it. They then show that decoder-only transformers exhibit what is known in graph network theory as "over-squashing", which may cause loss of information in the final prediction.

Strengths

The paper highlights a curious phenomenon in current language models, which shows their limitations even for extremely simple tasks like counting. The paper is well organized and well written. The observation of the over-squashing phenomenon is interesting.

Weaknesses

The observed phenomenon, while interesting, has limited scope. Also, the representational collapse result is hardly surprising to me. It feels quite trivial that in the limit of a very long sequence, the importance of a single token out of many repeated ones becomes negligible.

Questions

1) While the paper claims that the transformer architecture that they analyze is the most popular in state-of-the-art LLMs, it seems to me that they only consider relative positional embeddings and not absolute ones. To the best of my knowledge, current LLMs like GPT and Llama all use absolute positional embeddings. Would your results also extend to this case? In any case, I would highlight the use of relative positional embeddings more prominently in the main text. 2) In the actual prompts used for the experiments, the sequence does not appear at the end of the prompt but somewhere in the middle. Would your theoretical results somehow generalize if the repeated token is not the last one? Would anything change in the experiments if the sequence appears exactly at the end?

Rating

6

Confidence

3

Soundness

3

Presentation

4

Contribution

2

Limitations

N/A

Thank you!

We would like to thank you for acknowledging our responses and increasing your score. We remain available should you have any additional questions during the discussion period!