Summary
State-of-the-art language models (LMs) are now used to perform tasks (such as, e.g., question answering) with no fine-tuning; these models are fed the question and target options in their context, and their next-token distribution is then used to select an answer from this set of options.
LM outputs, however, are known to vary significantly based on the order in which these options are fed to it.
This paper proposes set-based prompting, a method to make LMs invariant to such option reorderings.
In short, the method feeds all options to a transformer model “in parallel” by modifying their position indexes to start from the same value, and by masking the attention so that options cannot attend to each other.
In practice, this makes the model run slightly out of distribution, as tokens presented to the model after the QA options will be able to attend to multiple tokens with the exact same position (as the multiple options are assigned this equivalent position indexes).
The paper has experiments on two tasks, question answering and language understanding.
Strengths
The paper presents a simple solution to a well known problem in NLP.
The paper runs experiments with a reasonable number of language model families: gpt-2, llama2, llama3, and mistral.
Weaknesses
In my opinion, this paper presents what is a relatively simple solution as being more complex than it is. (To be clear, I think this simplicity is one of the solution’s positive aspects, and not a negative aspect. Its presentation, however, could be considerably simpler.) Further, the proof of order invariance is sold as an important contribution, but it is also relatively straightforward. Since the methodological/theoretical contributions of this paper are relatively minor, though, I would expect it to have more experiments to confirm its efficacy. The paper, however, only presents two experiments, which are not very comprehensive in assessing the method's performance. More details below.
* The paper’s contribution is a simple (yet potentially efficient) method to make LMs invariant to prompt ordering. This solution amounts to reindexing token positions and slightly changing the model’s attention mask. However, the paper takes almost 6 pages to present this relatively simple solution.
* The paper only runs experiments on two tasks, and with one dataset each. This is not a comprehensive evaluation. The paper could run experiments with other tasks. Specifically, analysing the performance of this method on long-context information retrieval, as in Liu et al. (2024) could be interesting.
* In-context learning is another important setting in which models are susceptible to input order variation. However, the paper only focuses its experiments on zero-shot experiments.
Liu et al. (2024). Lost in the middle: How language models use long contexts
------------
Edit after author responses:
I thank the authors for their response, and apologise for only replying to it after the discussion deadline. While I still think that some aspects of the paper could be significantly improved, e.g., simplifying the presentation, the newly added experiments make the paper's evaluation much more comprehensible, so I am raising my scores (from 3 to 5). The experiments showing the performance degradation on the long-context experiment are quite interesting (even if they present negative results), and experiments with ICL show another important setting for which this method can be useful.
Questions
I don't have any specific questions. But as a suggestion, I think the stacked box plots in Figs 3 and 5 are quite hard to read. It would likely be easier to read them if the different conditions were presented as separate bars.
Limitations
The authors mention that their solution will make models run slightly out of distribution. Expanding on this point could be interesting, maybe discussing possible ways to mitigate this issue, such as fine-tuning models.