An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding

Summary

The paper introduces CREAM (Continuity-Relativity indExing with gAussian Middle), an efficient method for extending the context window of large language models (LLMs) to handle longer contexts without the need for extensive fine-tuning. CREAM manipulates position indices for shorter sequences within the pre-trained context window, using two key strategies: continuity, and relativity. Additionally, CREAM employs a truncated Gaussian distribution to enhance the model's focus on the middle part of the context, mitigating the "Lost-in-the-Middle" problem. Comprehensive experiments demonstrate CREAM's efficiency and effectiveness, showing improved performance over baselines like RandPos and PoSE on various tasks.

Strengths

CREAM achieves superior performance across a spectrum of long-context tasks. The empirical results substantiate CREAM's ability to effectively mitigate the "Lost-in-the-Middle" problem.

Weaknesses

I think the presentation of this paper is a significant problem. Actually, I cannot understand the method itself nor the intuitions behind it. First, I do not understand the concept of continuity in positional encoding. The authors use a short paragraph to explain continuity (lines 69-74) without any formulation, which is very abstract. The same goes for relativity, though I understood it from another paper. After explaining these two concepts, you still need to explain why they are important (I did not see this in the paper). Second, I do not understand why splitting the pre-trained context into three segments with different sizes can achieve continuity and relativity. Third, I do not understand why truncated Gaussian middle sampling mitigates the "Lost-in-the-Middle" problem. Line 113 states that it reduces the interval overlap in Eq2. But what is interval overlap and why does it result in the "Lost-in-the-Middle" problem? I may not be the ideal reader for this paper, but if other reviewers feel the same way, this paper may have been hastily completed.

Questions

See Weaknesses

Rating

3

Confidence

2

Soundness

2

Presentation

1

Contribution

2

Limitations

None

Summary

The paper presents CREAM (Continuity-Relativity indExing with gAussian Middle), an innovative approach to extend the context window of Large Language Models (LLMs) without the need for extensive fine-tuning at the target length. The authors address the "Lost in the Middle" problem, which plagues long-context LLMs by causing a performance drop when retrieving information from the middle of the context. CREAM achieves this by manipulating position indices and introducing a truncated Gaussian sampling method to focus on the middle part of the context during fine-tuning.

Strengths

1. **Novel Approach**: CREAM offers a novel positional encoding strategy that efficiently extends the context window of LLMs, which is a significant contribution to the field. 2. **Empirical Evidence**: The paper provides strong empirical evidence through comprehensive experiments, demonstrating CREAM's superiority over existing methods like PoSE and RandPos, especially in handling middle context information. 3. **Training Efficiency**: The method requires fine-tuning at the pre-trained context window size, which is computationally efficient compared to fine-tuning at the target length. 4. **Theoretical Foundation**: The paper includes theoretical justifications for the use of truncated Gaussian distribution, adding rigor to the proposed method.

Weaknesses

1. **Generalizability**: While the paper shows impressive results, it is not clear how generalizable these findings are to other LLMs beyond the tested Llama2-7B model. And I'm wondering whether the method can work with PEFT tuning techniques, such as QLoRA. 2. **Lack of Comparative Analysis**: The paper could benefit from a more detailed comparative analysis with other contemporary methods. For example, in Table 2, the paper only compares one model (LongChat-v1.5-7B-32k) with the same 32k context window. I suggest that in such comparision, peer models should at least take the same length of input, otherwise, it cannot prove the effectiveness of the proposed model over other Long LLMs with 32k+ context windows. I suggest the author compare with models like Mistral-7B-Instruct-v0.2, LongLoRA etc.

Questions

1. What are the computational overheads associated with implementing CREAM, and how does it scale with larger context sizes? 2. Are there any specific hyperparameter settings in CREAM that are particularly sensitive, and how were these chosen?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The discussion in Appendix F regarding the limitations of the CREAM method could be misleading. The author implies that CREAM provides enhanced performance compared to other methods concerning the "Lost in the Middle" issue but acknowledges that the problem is not fully solved due to the nature of decoder-only models. For a genuine limitations section, it is essential to delve deeper into the specific constraints of the CREAM approach and how these limitations might impact its application and effectiveness. A more explicit acknowledgment of the method's shortcomings and a clear explanation of why these issues cannot be overcome with the current model design would significantly improve the section's clarity and usefulness.

Summary

The paper proposes a new method, CREAM, that better enables extrapolation to longer contexts via finetuning at the base context length. The method uses positional embedding interpolation with the embeddings divided into three areas of interest and uses a truncated Gaussian to sample the position used for the middle segment, to increase training of the middle context. They show that this is at least as effective as prior length extrapolation methods that use finetuning at the base context length, and that CREAM-trained models suffer less from the lost in the middle problem.

Strengths

S1. The method shows clear improvement on the targeted issue (the lost in the middle phenomenon)-- e.g. in Table 1. S2. The method is well-motivated both intuitively and theoretically, and the explanation in section 2.2 is well-written and easy to follow. S3. The authors evaluate against the two most reasonable baselines (to my knowledge) and across a good selection of synthetic, long-context, and short-context tasks. (It would be an added benefit to evaluate against finetuning with position interpolation and *long* context at finetuning-time, to see the efficiency-performance tradeoff; however, I do not think this is strictly necessary to the paper's claims.)

Weaknesses

W1. The method is fairly similar to POSE in concept and performance. While I think the idea of emphasizing middle training is reasonable, well-explored, and clearly effective (e.g. in Figure 1), this has also been explored as a post-training or data selection correction in other work. Given the slight performance degradation from POSE on short-context tasks (i.e. in Table 4), I'm concerned that this might not be worth the tradeoff practically to do at training time. W2. Evaluation settings can differ subtly across papers, and so it would be better for the authors to reproduce baseline results rather than citing them from literature, where possible. This is most critical, I think, in Table 4, where the differences between methods are relatively small and all results are compared to the same baseline method, which is not an infeasibly expensive or unavailable model to run.

Questions

Q1. In Table 5: why does the 256k-extrapolated version of CREAM underperform the 128k-extrapolated version on short contexts? Q2. Say you had an even more limited amount of compute, to the point where finetuning on the base context length was prohibitively expensive (e.g., maybe you want to finetuned Llama3-8b but can't fit 8192 context at training time). Do you believe CREAM would be useful in this setting? Do you have any results (or speculation!) that suggest when the method may begin to break down with decreasing maximum training length? Q3. (Minor) What is the context length used in Figure 1? The x axis only reports the position of keys. Suggestions/comments: * Please explain in the text what the occasional highlighting of numbers represents (e.g. in Table 2). It would be helpful for readability to bold the best number in each column across tables.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes, no concerns here.

Summary

The paper proposes a method for extending the context window of pretrain large language models. The approach, CREAM, relies on modifying position indices to interpolate the positional encodings. Despite the often computationally expensive nature of such work, CREAM can extend to very long context windows while only needing to train at the original pretrained context window. Additionally, in their approach to the context length problem, the authors propose a solution that focuses explicitly on learning at the middle of the context — a span that often under-performs in long context models. The experimental results cover a wide variety of problems, both in terms of tasks, as well as context length challenges. Overall the results look very promising, and show a strong method for addressing a challenging task of extending the context length of LLMs.

Strengths

- The authors tackle two problems that go hand in hand, namely extending the context length during fine-tuning - but doing so in a way ensures consistent performance. - The approach divides the desired context length into three segments, which then results in relative positional distances that vary and consequently learning all relative positions within the target length $L$. The technique is simple, but clever and an effective way to efficiently expose the model to a broader range of relative positional distances during training. - The results are very strong, particularly with the performances listed on Long Bench, which encompasses a broad range of tasks.

Weaknesses

- No error bars shown on results. In most cases the results are quite strong, but the error bars would be helpful — particularly in some of the closer comparisons with PoSE. - The solution of using a truncated Gaussian approach lacks motivation. The "lost in the middle" problem is clear, however the solution of using a truncated Gaussian to force more focus on the middle seems brittle. Could the parameters of the truncated Gaussian be learned from data? The approach works well based on the results shown in Figure 5a, however.

Questions

The authors mention that CREAM only needed to be trained for 100 in some cases. Can the authors provide more insight on why the results are very good despite so little training? This seems like a significant achievement, but was not discussed in detail.

Rating

6

Confidence

3

Soundness

2

Presentation

4

Contribution

4

Limitations

The amount of focus on the middle context seems fixed in this approach. While the results for Gaussian truncation appear promising, it's not clear if this direct approach to solving the "lost in the middle" problem is universal. In short, how do we know how much to reweight the importance of the middle context, and does it vary by dataset or task?