Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding

Summary

Paper addresses the lost-in-the-middle effect, observed in the past for some LLMs, with a method called Multi-scale Positional Encoding (Ms-PoE) where position is encoded using different scales for each attention head. More precisely, for each head a re-scaling ratio is substituting the position m of a token with m/r (r can be 1.5 for instance) and different values of r are used for each attention head. This multi-scale view is evaluated experimentally and authors claim that this mitigates the ‘lost-in-the-middle’ effect.

Strengths

-a new approach for positional encoding Multi-scale Positional Encoding (Ms-PoE) -approach is plug-and-play (no additional fine-tuning needed)

Weaknesses

-The "lost-in-the-middle" effect is often assumed to impact all LLMs. But recent research indicates that this effect does not uniformly affect all LLMs, though the effect was observed on models such as Vicuna7B. see for instance https://arxiv.org/abs/2403.20262 which shows that not all LLMs display the lost-in-the-middle effect on a long context benchmark -For evaluation, the ZeroSCROLLS benchmark was used but authors could have considered incorporating other benchmarks designed for long context scenarios, such as "needle in the haystack" tasks. There are also many other relevant benchmarks available, including: NarrativeQA; LongEval; LongBench; LongBench-Chat; Loogle; ∞Bench and Long Range Arena (LRA) among others => on that aspect paper could have been stronger experimentally if evaluating the approach on more long context benchmarks

Questions

-eq(2): how are Rmin & Rmax fixed ? -fig 5: why only vicuna model evaluated, does Llama-2-7B-Chat also display a lost-in-the-middle effect ? -tab.3 are the differences between begin/middle/end results really significant ? A statistical test (for instance one-tailed Welch’s t-test) would have been welcome here

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

No 'limitations' section was provided

Summary

This paper proposes a plug-and-play method named Ms-PoE to mitigate the lost-in-middle challenge of LLMs. Specifically, Ms-PoE leverages multi-scale position embeddings to enhance information awareness in different parts of the context. Without fine-tuning the model, Ms-PoE achieves an average accuracy gain of up to 3.8 on the Zero-SCROLLS benchmark over the original LLMs.

Strengths

1. MS-PoE is a plug-and-play, simple yet efficient method, which is training-free and can achieve good performance. 2. The observation of "position-aware" attention heads is insightful, and applying re-scaling ratio dynamically to each attention head is reasonable. 3. I strongly agree with the authors' opinion that "the persistent difficulty faced by most LLMs in identifying relevant information situated in the middle of the context has not been adequately tackled."

Weaknesses

1. It seems that the performance improvement on MDAQ and ZeroSCROLLS benchmark is small compared to the "Self-Extend" method, which can be seen as a special type of "Multi-Scale Positional Encoding", i.e., "Single-Scale Positional Encoding". 2. Although this paper suggests that Ms-PoE is suitable for "Long Contexts," I believe it is suitable for any context situation. What exactly defines a "Long Context"? Is it a context length above 12K? Additionally, I have not come across any analysis of the impact of context length and how Ms-PoE can help the model understand the middle of long contexts. It might be beneficial to conduct experiments to determine whether Ms-PoE can enhance the model's understanding of the intermediate content of the context. This could involve analyzing different context lengths, such as inserting a specific length of context and evaluating the model's predictions.

Questions

1. What's the performance of MS-PoE(with LLama-2-7B-Chat as the backbone model) on LongBench-EN benchmark ? 2. Why chose 1.2 and 1.8 for R_{min} and R_{max} in Equation 2 ? 3. See Weakness 2

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

See Question and Weakness above.

Summary

This paper addresses the 'lost-in-the-middle' issue in large language models (LLMs) by introducing Multi-scale Positional Encoding (Ms-PoE). This approach enhances LLMs' ability to handle relevant information in the middle of the context without fine-tuning or added overhead. Ms-PoE uses position index rescaling and distinct scaling ratios for different attention heads to maintain essential knowledge. Experiments show Ms-PoE improves LLM accuracy, achieving up to a 3.8% gain on the Zero-SCROLLS benchmark.

Strengths

1. This paper is well-written and very easy to follow. 2. The analysis of the position embedding sensitivity of different attention heads is very interesting and intriguing.

Weaknesses

1． The improvement is actually limited, despite the method’s efficiency and ease of use. 2． The heuristic of the scaling ratio allocation process is a little arbitrary. There could be better solutions than just using an arithmetic sequence to achieve that. It is not strictly proved that a larger S_P means that the model should have a less rescaling ratio and vice versa. There could be possibilities that different heads could use the same scaling ratio or the other way around. 3． Moreover, there is no evidence that the rescaled position embedding could make the attention heads like the “Top in Figure 4” better at catching relevant information except for an empirical marginal task performance improvement and the comparison with other strategies in Section 4.3. An attention map of the attention heads after the rescaling could better strengthen this main claim of the paper. The authors could also partially show this by examining the S_P variation before and after the rescaling. 4． In Section 3.1, directly adjusting the hyperparameter and seeing the results on the test set is not very appropriate since it means the test set information is leaked. The authors should analyze the scaling ratio on the validation set and see if it is consistent with that in the test set. Moreover, the selection of the hypermeters R_min and R_max is also probably acquired by the test set performance, reflected by the ablation experiments in Table 4.

Questions

1. L212-215: “With a small scaling ratio LLMs tend to focus more on the most recent part of the context while with a large ratio, LLMs favor the beginning part” Are there any supporting results for this claim? 2. Did you choose the hyperparameters of R_min and R_max directly from their performance on the test set? 3. I’m happy to raise the scores if all the concerns are well addressed.

Rating

5

Confidence

4

Soundness

3

Presentation

4

Contribution

2

Limitations

N/A

Respond to authors

Thanks for your response. Although Ms-PoE is an efficient and plug-and-play method, it may require some parameter-selection processes and the hyper-parameter may vary for different situations. I want to understand the impact of R_min and R_max. Do these two values significantly affect the final results?

Response to Reviewer LVQT

Dear Reviewer LVQT, We sincerely thank you for your time and effort in reviewing our work. Your constructive and insightful feedback has been invaluable in helping us improve the quality of our manuscript. We have carefully addressed each of your comments and hope that we have successfully resolved all of your concerns. We are open to further discussion if you have any additional questions and look forward to your response. Best regards, Authors