Summary
The paper investigates the role of RoPE in long-context LLMs. It highlights that the base of RoPE crucially affects the model's ability to handle long contexts. This paper derives a theoretical (and empirical) lower bound for the base value required to maintain long-context capabilities and validates this through empirical experiments with models like Llama2-7B and Baichuan2-7B, and a 2B model trained from scratch. This work offers insights into the base of RoPE for long-context processing in LLMs, which is inspiring for the development and design of long-context LLMs.
Strengths
1. The lack of long-context ability is still an under-explored question and a very important one. Thus, the research topic of this paper is both theoretically important and practical.
2. The relationship between the RoPE base and long-context ability is important, and the proposed lower bound is interesting and inspiring.
3. The experiments in this paper are comprehensive and well support the claims made in the paper.
4. The presentation of this paper is overall very good and easy to understand and follow.
5. The claims of this paper are inspiring for the development of long-context LLM.
6. I am also very interested in RoPE-based selection in LLM and like this paper.
Weaknesses
1. The Desiderata 2. The similar token gets more attention in Section 4 seems intuitively correct but may not be empirically correct. As this desideratum is the fundamental motivation of Theorem 1, a thorough empirical verification is a must-have.
2. The motivation section is not well-written or organized. It confuses me while I read this part. These are just some previous empirical observations and are not deeply discussed. As the motivation part is very important, I would like to see a clear and well-organized motivation.
3. For the Caption of Figure 6, it would be better to show how to derive the value of the dotted line. I would also like to see a detailed derivation here.
4. Will the author plan to release the models (including the fine-tuned Llama2, Baichun2 with varying lengths, and the 2b model series)? This would benefit future work.
5. I wonder about the negativeness and positiveness of the $q^{T}R_{m,\theta}k$ in equations (8) and (9). If the values are negative, say -2 and -1, which one indicates more attention?
6. For section 5.3, I would like to regard this as a conjecture rather than an explanation.
7. Moinries
- The upper case and lower case of the title are not consistent, such as the title of Section 4.1 and Section 2.2
- Line 225 “method2” -> “Method 2”