Length generalization, the ability to generalize from small training context sizes to larger ones, is a critical challenge in the development of Transformer-based language models. Positional encoding (PE) has been identified as a major factor influencing length generalization, but the exact impact of different PE schemes on extrapolation in downstream tasks remains unclear. In this paper, we conduct a systematic empirical study comparing the length generalization performance of decoder-only Transformers with five different position encoding approaches including Absolute Position Embedding (APE), T5's Relative PE, ALiBi, and Rotary, in addition to Transformers without positional encoding (NoPE). Our evaluation encompasses a battery of reasoning and mathematical tasks. Our findings reveal that the most commonly used positional encoding methods, such as ALiBi, Rotary, and APE, are not well suited for length generalization in downstream tasks. More importantly, NoPE outperforms other explicit positional encoding methods while requiring no additional computation. We theoretically demonstrate that NoPE can represent both absolute and relative PEs, but when trained with SGD, it mostly resembles T5's relative PE attention patterns. Finally, we find that scratchpad is not always helpful to solve length generalization and its format highly impacts the model's performance. Overall, our work suggests that explicit position embeddings are not essential for decoder-only Transformers to generalize well to longer sequences.
Paper
Similar papers
Peer review
Summary
Beyond model training, sequence length is one of the crucial aspect impacting the fluency, accuracy and completeness of LM outputs. For transformer based model's positional embeddings (PE) are used to explicitly represent the location of tokens. Over the past few years several works have attempted to propose a better PE approach, particularly aiming to improve over the original absolute PE of the Transformer. This work, proposes a no positional encoding (noPE) model training for decoder only Transformer, to better generalize on generation length. The proposed noPE LM is evaluated on set of maths and reasoning tasks, using a small size (~100M parameter) LM. Findings show that noPE based LM performs similarly with relative PE based model (from the T5 model), and outperforms models with alternative PE variants such as Rotary, AliBi, and absolute PE for length generalization. Along with the evaluation results, detailed empirical analysis and theoretical arguments are provided for why noPE LM can intrinsically support an absolute and relative token position representations.
Strengths
This work, provided an in-depth analysis of Transformer LM training and inference without using PE. Although, the focus is to address the question of LM length generalization, the findings inform future possibility of an increasingly simplified transformer based LM's. In addition to the surprising results where noPE works in a similar way as the best performing relative PE (T5's PE), the work, delivered a thorough discussion as to why noPE works.
Weaknesses
Despite the finding not using PE (noPE) can generalize for sequence length, the following points remain unanswered or not fully investigated: - Tasks selected for modeling and evaluation does not cover challenging length generalization scenarios. For instance, the noPE approach could have been testing using a summarization task which is highly impacted by the length of the summary. - Regardless of the provided results on length generalization, detailed and task specific performance metrics comparing LM's with PE's and noPE LM is missing. The absence of these comparison leaves open question on task specific performance comparison.
Questions
- In addition to the mean rank, can authors add more task specific evaluation metrics, or could you please provide more motivation on why not to report task specific performance metric? - This can be a hypothetical one, have you considered a fine-tuning approach for LM's, for instance removing the PE when fine-tuning for down-stream tasks? - Can authors expand more on the selection of the LM size (100M parameters)? can we expect similar findings to hold if model size changes (i.e. larger LM's with more diverse and large scale train data)?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
Limitation is provided, specifically explaining why the proposed noPE is not applied for large scale LM training. The limitation, not being able to access publicly available LM can be further expanded, for instance why not use 500M? If there is any identified societal impact its encouraged to include it.
Summary
This work follows a long line of work analysing the effectiveness of various types of positional encodings and, in particular, a line of work exploring the effectiveness of causal masking without any positional encodings [1 - 8]. To that end, they evaluate the length generalisation performance of Transformers trained with various types of positional encoding schemes on a diverse set of reasoning and algorithmic tasks and find that Transformers without positional encoding schemes generalise better to higher lengths than other encoding schemes. They consider 3 three types of tasks -- primitive (copying or reversing strings), algorithmic (addition, parity, etc.), and prior length generalisation benchmarks (e.g. SCAN). They evaluate five different encoding schemes, including Alibi and rotary encodings. They use autoregressive decoder-only Transformers of sizes of the order of 100M. For each task, Transformers are first trained on a set of examples using a language modelling objective and are then evaluated on examples of higher lengths. Across the three types of tasks, they find that Transformers without positional encoding (but with masking) and T5's relative bias outperform other encoding schemes. Additionally, they show via a construction that Transformers without positional encodings can simulate both absolute and relative encoding schemes. They compare the attention patterns of Transformers trained without positional encoding schemes with other schemes and find that the attention patterns are most similar to relative positional encodings. Lastly, they find that when trained with a scratchpad, T5's relative positional encoding scheme and no positional encodings seem more effective than others.
Strengths
(S1) **Interesting results.** Overall, I believe there are several interesting findings in the paper. I believe the contributions in the paper take a step forward in understanding the effectiveness of Transformers with just causal masking. The analysis with the scratchpad setup demonstrating the importance of PE for scratchpad and the effect of scratchpad for length generalisation is interesting as well. (S2) **Sound experimental setup and well-motivated.** The question of understanding and identifying the most effective positional encoding scheme is an important problem, and this work takes a step forward in that direction. The choice of tasks seems interesting, and the evaluation with Transformers trained with language modelling objectives is more aligned with practice than prior works that have explored this direction. (S3) **Well written.** The paper is well-written, the motivations are clearly outlined, and the arguments are well-presented.
Weaknesses
(W1) **Novelty of results is slightly overclaimed.** Based on my limited knowledge, the use of Transformers without positional embeddings and with only causal masking was first studied in [1]. It was later analysed empirically [2,3] and theoretically [3, 5] in further works. These works mostly suggested that Transformers without positional encodings are competitive with standard encoding schemes on machine translation and language modelling tasks. On synthetic formal language tasks, [4] and more recently [6] have noted that using no positional encoding leads to better length generalisation performance. In more practical settings, recently [7, 8] have shown that large language models trained with only causal masking are competitive with models with positional encoding. The fact that Transformers with only causal masking can encode positional information and generalise well to higher lengths is not entirely new. To be fair to the authors, I do not think that they explicitly claim this. I think the paper does a good job of establishing that just causal masking is better at length generalisation than various other schemes in a more systematic way. However, given the prior work exploring Transformers with only causal masking, the framing of the introduction seems a bit odd, and some of the papers mentioned above [1-8] are mentioned passively in the paper. It could be helpful to readers unfamiliar with the area if the prior work was presented in a more comprehensive manner. (W2) **Effect of scale.** Since the tasks in the paper are confined to synthetic tasks and models with ~100M parameters, it is still unclear if training LLMs without positional encodings will be better for length generalisation in practice. Modifications to Transformers at this scale for effectiveness on such tasks have often not scaled to LLMs in practice. As mentioned in the paper, that could require a larger computational budget which is understandable to some extent. But at the same time, comparing Transformers trained for language modelling on text data on higher lengths would have been useful. Based on the results in [7, 8], it seems like positional encoding schemes help improve perplexity in language modelling, but it may be interesting to see if that changes for length generalisation tasks. [1] DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding. 2017 [2] Assessing the Ability of Self-Attention Networks to Learn Word Order. 2019 [3] Transformer Dissection: An Unified Understanding for Transformer's Attention via the Lens of Kernel. 2019 [4] On the Ability and Limitations of Transformers to Recognise Formal Languages. 2020 [5] On the Computational Power of Transformers and its Implications in Sequence Modeling. 2020 [6] Transformers Learn Shortcuts to Automata. 2022 [7] Transformer Language Models without Positional Encodings Still Learn Positional Information. 2022 [8] Bloom: A parameter open-access multilingual language model. 2022
Questions
(Q1) Regarding Section 5.2, if the models are trained with different positional encoding schemes from scratch, how can similarity between attention patterns help us determine how the model is computing positional information? Is it not possible that the models have very different learning trajectories? What about cases when Transformer without positional encoding performs better than relative encodings, wouldn't the underlying mechanism be different in that scenario? I think it is interesting the attention patterns without encodings are relatively much closer to ones with relative encodings (close to the baseline with different initialisations) than other ones, and that does provide some evidence that it is closer to relative encodings. But it is not clear to me why comparing the attention patterns should help us determine which type of positional information is being used.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
NA
Summary
Length generalization is one of hot topics for transformers. In this paper authors investigate how positional embedding influences this generalization for **decoder-only** models on **reasoning tasks**. Authors compare for variety of tasks (e.g. copy, summation, sorting, etc.) sinusoidal positional embedding, T5, AliBi, Rotary and no positional embedding (NoPE). First, authors proof by construction that NoPE for decoder-only models is able to learn positional information due to causal masking. Then authors show that none of the positional embeddings generalizes to longer sequences, while NoPE in some cases performs even better than others (though performance is still bad compared to seen lengths). Empirically by comparing attention patterns authors show that NoPE learns attention similar to T5 models. Finally authors also play with scratchpad and show that it is not always helping generalization for longer sequences.
Strengths
- first time (at least I didn't see in prior works) formal proofs that decoder-only models are capable of positional embedding reconstruction (I went through the proof, no issue found with the math) - emergency for other positional embeddings development or investigation of transformers failure for the longer sequence generalization for reasoning tasks - interesting empirical results for reasoning tasks showing that mainly all positional embeddings currently widely used do not work for reasoning task on longer sequences while no positional embedding (NoPE) works similarly or even better sometimes - interesting insights on the NoPE (learnt attention pattern is more similar to models trained with T5 bias) and results on scratchpad
Weaknesses
- overstatement in the test about positional embeddings for the decoder-only models: here it should be clarified about the type of the tasks where we apply, as for LMs we know that e.g. AliBi is working. - small scale experiments and limited length generalization evaluation. - no in-depth comparison of the root for the different behaviour e.g. for AliBi between the reasoning tasks and LM task.
Questions
- from the Abstract and introduction I expected to see NoPE working well for GPT-models, while main results are provided for small-scale models and reasoning tasks rather than language modeling itself. I guess could be better to state about reasoning tasks in the abstract in addition to the restriction to the decoder-only models to be considered - From results (Appendix plots) I can see that actually no any positional embedding (including NoPE) generalized to long sequences for the reasoning tasks, so the statement "Overall, our work suggests that explicit position embeddings are not essential for decoder-only Transformers to generalize well to longer sequences." is not supported as NoPE also doesn't work (works only for very couple cases + sequence length is not very large compared to training one). - For RoPE did authors try to insert it in every layer similar to T5 or AliBi which are used in every transformer layer? Otherwise could be not fair comparison in the end if the positional embedding saturates over the layers. - Eq. (2) why do we take min? Also do we consider all possible pairs of attention heads or we do pairing head1 from A and head1 from B? For other part of evaluation I found smart to use another seed with NoPE to measure how it changes to see that it actually like T5. - line 40-49: even for relative positional embedding it was shown that it could fail to generalize, see "CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings", NeurIPS 2021. In the same paper as well as e.g. in "An image is worth 16x16 words: Transformers for image recognition at scale", ICLR 2021 "Conditional Positional Encodings for Vision Transformers", ICLR 2023 there were reported that NoPE can works for the encoder-based models. - line 301-305: yes, authors consider LLMs style models (which is just decoders) but the data / tasks are different, so that we cannot just generalize from the observations to the LM data as in LM data we saw opposite behaviour. I found confusing the whole paragraph in this context. - Limitations stated as "not study how large-scale pretraining affects" - AliBi did study on WikiText103 which not large-scale + model was also rather small <1B params. I believe the issue is the data / task type (see below comments). - Notation of multiplication as dot is weird. Prefer to see normal multiplication in $k.d$ -> $k\cdot d$ - Eq. 29 $W_V$ -> $W_K$ - Fig. E2 for Copy Variant 1: did authors check if bias in T5 is learnt to be negative or positive? As in AliBi it emulate smoothed window attention and far away positions are mainly ignored, while for the task if copy you need to look at the whole sequence indeed. So could be that design of the positional embeddings are not designed for specific tasks where another reasoning on the positions is done (e.g. for LM we just need some summary context, but not exact values for every positions as we don't do e.g. copy task). Wonder to see what T5 bias then learnt to be consistent or disprove this hypothesis. For now the hypothesis seems to be consistent along all plots shown in the Appendix: e.g. for all copy tasks and reverse AliBi doens't work); for sorting Rotary is the best - which can be explained as we can solve sort by just comparing neighbors and reordering them, so strong relative position bias is needed; for other more or less different positional embeddings behave similarly. - Why do authors consider only 2L sequence generalization and not longer as in AliBi? I see that in the end we are not able to generalize even from 20 positions to 40 positions for simple reasoning tasks, while in LM (e.g. AliBi) they generalize to longer contexts, e.g. 16k tokens. I think in the main text it is overstated that some positional embeddings do not work - there should be clearly expressed "let's consider reasoning tasks different from LM and investigate how there length generalization happens." I find that it would be interesting to have discussion on the difference between LM task and considered in the paper, as right now for me it is not clear why LM generalization with AliBI e.g. works nicely for even 16k tokens while in the simpler reasoning tasks this is not the case. Is it model? data? task itself? I presume we have more complex dependence on the positions and sequence length than in standard LM where simple summarization of far away context could be enough for good generation longer sequences while in reasoning task it is crucially to have proper positions distinguishing.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
4 excellent
Limitations
Limitations in the main text, however main restriction on considering non-LM tasks is not clearly delivered in the paper. **Update: Originally score was 3 for contribution and overall score 6-weak accept. After rebuttal and discussion they are raised to 4 for contribution and 7-accept.**
Summary
The paper presents a comprehensive study on the role of positional encodings in Transformer models, focusing on their ability to generalize over sequence lengths. The authors introduce a novel positional encoding method, which does not use explicit positional information.
Strengths
1. The paper introduces a novel positional encoding method, which is a significant contribution to the field. 2. The paper is well-written and easy to follow.
Weaknesses
The authors did not study how large-scale pretraining affects different PEs due to the lack of publicly available large language models trained with various PEs under similar conditions. This leaves a significant aspect of the problem unexplored.
Questions
1. How would NoPE perform in a broader range of tasks beyond the mathematical and reasoning tasks evaluated in this study? 2. Could the authors elaborate on the potential implications of their findings for the design of future Transformer models?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
NA
Reviewer response to the rebuttal
Dear authors, Thanks for all the details and clarifications! I have carefully read all reviewers' comments and your responses. Let me add a bit more comments for the sake of community and fruitful discussion we started: > Clarifying the Scope and Generalization Setup After rereading the abstract and introduction I agree that it is more or less clearly stated in the paper. I would suggest maybe to highlight not only downstream tasks but also reasoning as itself. I think your findings are really very impactful in the direction of reasoning showing that all current solutions are worse than noPE and we did not make any progress even in simple reasoning tasks. Maybe in the discussion you can even add an emergency of new general architectures to be able to solve the problem as it could be that positional embedding is tight to the modeling itself to avoid shortcuts models are doing now. > Small scale & Length Generalization Evaluation I am totally fine with current models size and scale as you highlighted that the purpose is to show that for e.g. simple reasoning tasks any positional embedding doesn't work even for small length increase. So I don't think that absence of large scale is any significant downside of the paper. > Analysis on discrepancy in ALiBi’s performance in Language Modeling vs Downstream Task You have resolved my concern. What I suggest is to include our discussion into the paper, pointing on the choice of 2L as generalization setting, and also referencing to the works which show issue with AliBi as window attention and discussing reasoning tasks more carefully here (tasks which need to have full attention, e.g. Copy and Reverse). I think this discussion could emerge connection between people in reasoning domain and LLMs domain to collaborate deeper on understanding reasoning for LMs. > Inserting RoPE in every layer. Thanks for the details. I am good with your evaluation here. Thanks! > Evaluation on attention heads I am all good, thanks for the updated figure - good to see consistency here. > References to generalization in relative PEs in vision Transformers Overall across prior works I see visible distinction between encoder, decoder and encoder-decoder archs (not that we should use noPE encoder for text - yes, it is a bag of words which could not produce good embeddings). Would be nice to have some clarification in text on this highlighting that right now we don't have a clear picture for all of them under the same hypothesis. But this is a more philosophical discussion :) . > Fig. E2 for Copy Variant 1: did authors check if bias in T5 is learnt to be negative or positive? As in AliBi it emulates smoothed window attention and far away positions are mainly ignored, while for the task if copy you need to look at the whole sequence indeed. So it could be that the design of the positional embeddings are not designed for specific tasks where another reasoning on the positions is done (e.g. for LM we just need some summary context, but not exact values for every positions as we don't do e.g. copy task). Wonder to see what T5 bias then learnt to be consistent or disprove this hypothesis. For now the hypothesis seems to be consistent along all plots shown in the Appendix: e.g. for all copy tasks and reverse AliBi doens't work); for sorting Rotary is the best - which can be explained as we can solve sort by just comparing neighbors and reordering them, so strong relative position bias is needed; for other more or less different positional embeddings behave similarly. Any comment on my previous question regarding T5? Anything else maybe you see from its values?
Dear reviewer, We appreciate your thorough review of our paper and response. It's gratifying to know our discussion **addressed your queries** related to the “**Scope**”, “**ALiBi performance**”, “**Attention head evaluation**”, and “**Models’ scale**”. We're also heartened that you consider our findings to be “**very impactful**”. We will ensure these discussions are appropriately reflected in the camera-ready. > Values of T5 Relative Bias Below, we present our analysis on the T5 Relative Bias values in the decoder (in self-attention) of the pretrained HuggingFace model. (If you have a specific task in mind, we'd be happy to revisit this analysis). Please refer to Appendix B.2 for a detailed breakdown on T5: Notably, this analysis illustrates the learnt bias value $b_{\mathrm{bucket}}$ for each relative distance $\mathrm{bucket} \in [0, 31]$). (each attention head has a distinct set of such biases). Smaller buckets corresponds to shorter distances between tokens, while larger buckets corresponds to the longer distances. Bucket 31 is exclusively used for all distances $\ge$ 128 tokens. We observe two primary patterns in the pretrained 3b model (We found these patterns to be consistent across all sizes): **Type I** (As demonstrated for head 1 in the below table) 1. Decreasing but *positive* values for close buckets: [0, 5] 2. Decreasing but *negative* values for middle buckets [5, 20], i.e, values are becoming more negative. 2. Increasing values for buckets [20, 30]. 4. *A significant negative value* for bucket 31 (which corresponds to all relative distances $\ge$ 128 tokens). |$\mathrm{bucket}$|Relative Bias Value $b_{\mathrm{bucket}}$|ASCII Representation of Bias Value| |-|-|-| |0|2.5469|`-----/#----`| |1|4.9688|`-----/##---`| |2|1.6406|`-----/-----`| |3|0.0967|`-----/-----`| |4|-0.6484|`-----/-----`| |5|-1.2422|`-----/-----`| |6|-1.5703|`-----/-----`| |7|-1.8516|`-----/-----`| |8|-2.0469|`----#/-----`| |9|-2.2344|`----#/-----`| |10|-2.4062|`----#/-----`| |11|-2.3906|`----#/-----`| |12|-2.4062|`----#/-----`| |13|-2.5156|`----#/-----`| |14|-2.5938|`----#/-----`| |15|-2.5625|`----#/-----`| |16|-2.6094|`----#/-----`| |17|-2.6719|`----#/-----`| |18|-2.6562|`----#/-----`| |19|-2.6094|`----#/-----`| |20|-2.4688|`----#/-----`| |21|-2.3750|`----#/-----`| |22|-2.1719|`----#/-----`| |23|-1.9297|`-----/-----`| |24|-1.5938|`-----/-----`| |25|-1.2812|`-----/-----`| |26|-0.9141|`-----/-----`| |27|-0.5508|`-----/-----`| |28|0.2305|`-----/-----`| |29|0.5859|`-----/-----`| |30|0.9805|`-----/-----`| |31|**-27.7500**|`#####/-----`| **Type II** (As demonstrated for head 2 in the below table) The same pattern is observed across buckets [0, 30], but, in contrary, we observe *very large positive value* for bucket 31. |$\mathrm{bucket}$|Relative Bias Value $b_{\mathrm{bucket}}$|ASCII Representation of Bias Value| |-|-|-| | 0 | 2.5000 | `-----/#----` | | 1 | 2.5469 | `-----/#----` | | 2 | 1.5234 | `-----/-----` | | 3 | 0.7383 | `-----/-----` | | 4 | 0.1758 | `-----/-----` | | 5 | -0.2832 | `-----/-----` | | 6 | -0.8281 | `-----/-----` | | 7 | -1.1719 | `-----/-----` | | 8 | -1.4375 | `-----/-----` | | 9 | -1.8516 | `----#/-----` | | 10 | -1.9219 | `----#/-----` | | 11 | -2.1875 | `----#/-----` | | 12 | -2.3438 | `----#/-----` | | 13 | -2.2656 | `----#/-----` | | 14 | -2.4062 | `----#/-----` | | 15 | -2.5625 | `----#/-----` | | 16 | -2.5781 | `----#/-----` | | 17 | -2.6406 | `----#/-----` | | 18 | -2.7500 | `----#/-----` | | 19 | -2.8125 | `----#/-----` | | 20 | -2.7500 | `----#/-----` | | 21 | -2.5469 | `----#/-----` | | 22 | -2.4375 | `----#/-----` | | 23 | -2.2969 | `----#/-----` | | 24 | -2.0000 | `----#/-----` | | 25 | -1.6719 | `----#/-----` | | 26 | -1.4375 | `-----/-----` | | 27 | -0.9844 | `-----/-----` | | 28 | -0.2734 | `-----/-----` | | 29 | 0.0588 | `-----/-----` | | 30 | 0.5312 | `-----/-----` | | 31 | **7.7812** | `-----/#####` | Diving deeper into **bucket 31**, its value across all heads (coincidentally, this model has 32 heads) is: |Head|Relative Bias Value for bucket 31 ($b_{31}$)| |-|-| |1|-27.75| |2|7.78125| |3|-28.75| |4|-29.125| |5|-31.375| |6|-32.75| |7|28.0| |8|12.4375| |9|33.5| |10|-25.875| |11|-35.25| |12|26.25| |13|28.75| |14|-27.875| |15|-39.5| |16|-26.375| |17|69.5| |18|-30.125| |19|-34.75| |20|-26.5| |21|-30.375| |22|45.75| |23|-32.5| |24|38.75| |25|-28.0| |26|29.0| |27|16.5| |28|4.75| |29|-31.0| |30|43.0| |31|-31.875| |32|-22.5| Mean = -5.883, SD = 31.349 Our observations suggest that while T5 has heads that mask distant tokens (Type I), other heads attend to distant tokens when required (Type II). This nuanced distinction could potentially account for the performance differences between T5 and ALiBi, as T5 can learn to adaptively attend to distant tokens, in contrast to ALiBi, which always (yet smoothly) mask distant tokens. We thank you again for your time and feedback and hope our response answers your inquiry. Are there any remaining questions you would like us to answer?
Dear Reviewer, Thank you for your invaluable feedback and the insightful discussion initiated. We're curious to know if you've had a chance to go over our previous response. We hope it has addressed your question. As the discussion phase nears its end, we're eager to address any outstanding questions you may have, and to incorporate any additional feedback. We hope you kindly consider a renewed assessment of the paper, considering our recent exchanges. Once again, we appreciate your time and dedication to the review process.
Final comments and raising the score
Dear authors, Sorry for delay (busy week). I had a look at your analysis of T5 - this is amazing! I found it is very interesting that T5 is learning some heads to ignore distant tokens and some in contrary. Also this shows that something non-trivial happens with distant tokens for embeddings which do not do window attention. I am happy with the analysis you provided and strongly recommend (and hope) you to include it into Appendix for the final revision. As I said before I think the paper provides very important analysis and results to understand better transformers, reasoning tasks and generalization. I am raising my score. Thanks again for nice and productive discussion!
Dear Reviewer, We're truly grateful for your active participation during the rebuttal and discussion phases. Your feedback has been instrumental in enhancing the quality of our work. We will incorporate the changes promised in the final version.
Dear Reviewer, Thank you again for your time for reviewing our paper. We would appreciate knowing if you were able to look at the response we provided. With the discussion period nearly over, we would love to answer any remaining questions you may have.
Dear Reviewer, We appreciate the effort you’ve put into reviewing our paper and offering insightful feedback. We’re wondering if you’ve had a chance to go through our response. As the discussion period is nearly over, we’d love to address any lingering queries or concerns you may have.
Thank you for the clarifications. As mentioned, it would be helpful to discuss prior work more comprehensively. Regarding the effect of scale, even though it is quite important, I don't think it is completely fair to expect those experiments due to resource constraints. I had already given a positive score and would like to maintain my score.
Dear Reviewer, We truly appreciate your engagement during the rebuttal and discussion periods, we believe your invaluable input has improved our work. We will make sure to incorporate your feedback in the camera-ready.
Decision
Accept (poster)