Summary
While next word prediction produces task-general models that can do a wide variety of tasks when prompted, it is not an efficient formulation in that it requires very large models. On the other hand, fine-tuned models achieve higher performance at smaller sizes but are specific to a few tasks. This paper proposes text alignment as a middle ground that encompasses a wide range of tasks while allowing for smaller models than next word prediction. Concretely, they convert 28 datasets (encompassing tasks like entailment, IR, QA, coref, and consistency) into the text alignment format and fine-tune RoBERTa on them. The resulting model outperforms much larger models (that are instruction-finetuned), as well as RoBERTa with task-specific fine-tuning.
Strengths
(1) The idea of using text alignment as a task-general interface is interesting and seems useful for producing useful task-general models at smaller sizes.
(2) The experiments are extremely thorough, and the method performs well across the board.
(3) The paper is well-written and clear.
Weaknesses
While this section contains some suggested experiments, I support the acceptance of this paper regardless of whether or not they are run during the rebuttal period.
(1) While the paper frames alignment as being more general than multi-task finetuning, they only evaluate the model on tasks seen during alignment finetuning. Indeed, as shown in the task ablations in Table 5, it seems that the model can only do tasks that are included during training. Therefore, I wonder how different the model is from simply doing multi-task fine-tuning, and what clear advantages are provided by using a unified interface. While already strong, I think the results would be even stronger if there were examples of the model doing unseen tasks.
(2) Related to questions of how general alignment is, the tasks included in the training and evaluation feel very close to entailment, which might be a key factor in enabling them all to use the same interface without task-specific heads. Therefore, I wonder how performance would be affected if tasks very far from entailment were included. (For example, you could include POS tagging, where x_1 is the original sentence and x_2 is the sentence with some or all of the words replaced by their part of speech.)
(3) Related to the above, while the paper claims that changing the interface from next word prediction to alignment allows for smaller models, I wonder if the smaller model sizes are simply a result of considering a narrower set of tasks than the instruction-finetuned models. I suspect that FLAN-T5 needs larger model sizes simply because it needs more capacity to do more tasks. One relevant ablation testing this question would be to take the same RoBERTa model and instruction-finetune it on the same 28 datasets as the alignment model.
Questions
(1) For handling longer contexts, I wonder if it would make sense to take min_j max_i f(x_1^i, x_2^j) instead of mean_j max_i f(x_1^i, x_2^j) as in the paper, with the interpretation that all of the facts in x_2 should be supported. The mean could fail if x_2 contains mostly supported statements, except for an egregiously unsupported statement in a single chunk.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.