Cappy: Outperforming and Boosting Large Multi-Task LMs with a Small Scorer

Summary

The authors propose training a small(er) classifier model (‘Cappy’) to predict an answer given a set of possibilities (if the answer set is closed) or multiple generations from an LLM (in the case of open-ended/generative tasks). The model is trained on data from T0, using LLMs to generate partially correct responses alongside positive and negative examples. They evaluate performance on T0 evaluation tasks (classification) and big-bench (generative) and find that Cappy outperforms same-size models, and consistently provides improvements over a base model when picking from generations.

Strengths

The proposed approach is straightforward, using a trained classifier as a reranker for model generations. The improvements seem significant, and the fact that a classifier outperforms a similar-size generative model is interesting to see. The data gathering scheme for Cappy’s training data is interesting, and using rouge-l as a proxy for gold labels is interesting and seems to work well in ablations. The writing is clear and the paper is easy to follow.

Weaknesses

**Comparisons to Best-of-N** - I’m somewhat concerned about the novelty of this work. The proposed method is very similar to the best-of-N sampling originally proposed in [1] (“Best-of-N sampling”) and discussed (sometimes under ‘reranking’) in [2] [3], [9], inter alia. These methods work by sampling multiple LM generations and then picking the answer with the highest reward as chosen by a trained reward model. The three core differences between these prior works and Cappy I can detect are: 1. Different use in classification - Cappy directly uses the set of possible answers for classification tasks, while these prior works do not. 2. Training data - Cappy is trained on prompt source-based data with augmentation, which is interesting and seems to help performance, while (afaik) best-of-n uses trained reward models, which are often trained using preference datasets. 3. Application - best-of-n approaches are often applied and focussed on how they aid with aligning to human preferences, while the focus in this work is on improving over benchmarks. While these are interesting differences, the authors do not compare to these prior works and explicitly make these comparisons. At the very least, I would expect these methods to be discussed in a related work section, and the decisions made in training Cappy contrasted to them. Ideally, they would be integrated as baselines/ablations to justify differences, highlighting the merits of Cappy over these approaches and covering what is needed to adapt these approaches to Cappy’s chosen setting. **Missing prior work for reranking** - Cappy effectively is serving as a reranker in the generative case. There is a long history of work in answer reranking, that would be useful to discuss as related work (e.g., [4] for multi-choice qa, [5,7] for summarisation, [5,6] for open-domain qa). **Further baselines and ablations** - There are other methods for training and applying rerankers, including contrastive training (as used to train a reranker in [7], with rouge as a proxy for human judgements), or encoding multiple samples at once as opposed to one-at-a-time (as done in [4]). It would be useful to test these changes, especially given the authors suggest that making use of contrastive information is helpful for Cappy in line 202. It would also be interesting to compare against self-consistency [8], another approach for improving models that makes use of multiple generations. Given all of these things, I am inclined to recommend rejection for this paper in its current state. Further changes differentiating this approach against best-of-n sampling and further experiments exploring the full space of answer reranking would greatly improve this work, and may sway my opinion, but require significant extra work. I think the overall idea is interesting and evidently effective, but requires further work to make a substantial contribution. I hope the authors make these changes and are successful in the future! [1] Stiennon et al. (2020). Learning to summarize from human feedback. NeurIPS. https://arxiv.org/pdf/2009.01325.pdf [2] Bakker et al. (2022). Fine-tuning language models to find agreement among humans with diverse preferences. ArXiv. https://arxiv.org/pdf/2211.15006.pdf [3] Glaese et al. (2022). Improving alignment of dialogue agents via targeted human judgements. ArXiv. https://arxiv.org/abs/2209.14375 [4] Kratzwald et al. (2019). RankQA: Neural Question Answering with Answer Re-Ranking. ACL. http://aclanthology.lst.uni-saarland.de/P19-1611.pdf [5] Revaut et al. (2022). SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization. ACL. https://aclanthology.org/2022.acl-long.309.pdf [6] Lee et al. (2018). Ranking Paragraphs for Improving Answer Recall in Open-Domain Question Answering. EMNLP. https://aclanthology.org/D18-1053.pdf [7] Liu et al. (2021) SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization. ACL. https://aclanthology.org/2021.acl-short.135/ [8] Wang et al. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR. https://arxiv.org/pdf/2203.11171.pdf [9] Dubois et al. (2023). AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback. ArXiv. https://arxiv.org/abs/2305.14387 Edit: The authors have responded with some of the experiments and comparisons I asked for, with positive results, and so I'm happy to raise my score.

Questions

1. How is cappy different to best-of-n techniques proposed in the papers mentioned above? Do you think using preference data already available could help or augment your data collection strategy? 2. Did you explore contrastive objectives in pretraining cappy? How about encoding multiple answers at once? 3. Do you have further details on the distribution of rouge-l scores you used in the data augmentation setup? It would be interesting to see this, and do you think there are any limitations in using rouge-l against something like cosine similarity for this?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

I think the authors effectively discuss limitations. They miss some elements such as Cappy requiring multiple generations (thus increasing its inference cost over using a single-generation) and complicating the model pipeline, but cover Cappy’s weakness in the realm of complex logical problem-solving / mathematics, and its current reliance on supervised datasets.

Summary

This paper introduces an auxiliary module called Cappy, which aims to enhance the performance of large language models. Cappy operates by taking a task instruction and a proposed response as input, and it estimates the quality score for the response. The training process involves creating a dataset that combines correct and mismatched answers with various instructions, and the Cappy scorer is then trained using a regression objective. In downstream adaptation, the small scorer, Cappy, is fine-tuned on a small dataset that follows the same approach as the training set. During testing, predictions are determined by selecting the answer choice with the highest score. In essence, Cappy acts as a valuable filter that enhances the overall quality of generated responses.

Strengths

The idea of leveraging a lightweight scorer as a ‘’filter’’ is interesting. Fine-tuning the filters on downstream tasks can be efficient and practical.

Weaknesses

Cappy itself is only a filter for in-domain responses and can not impact the intrinsic ability of a pre-trained LLM. This method cannot handle tasks outside the LLM's expertise, emphasizing the need for a sufficiently general and powerful LLM. Alternatively, fine-tuning LLMs, as shown in Table 3, may be necessary but costly. By comparing able 2 and Table 3, fine-tuning is still very essential for task adaptation.

Questions

Please refer to the weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

No additional limitations.

Summary

Adapting SotA LLMs to novel tasks is difficult given their extreme size, yet is generally more effective than in-context learning alone. To address this, the authors propose “Cappy”, a scoring model (leveraging RoBERTa as a backbone) which is trained to score pairs of (instruction, response) for any task instruction and corresponding response. Cappy is pre-trained on many examples of ((instruction, response) -> score); after pre-training, Cappy can be easily fine-tuned (because of it’s small size) to a given task and then used as an augmentation to LLM generation by acting as a ranking mechanism atop several LLM generated responses. To create pre-training data for Cappy, the authors use the PromptSource dataset, training the model to predict 1 for correct pairs of (prompt, response) and 0 for mismatched pairs. Additionally, the authors generate additional responses using a large LLM (e.g. FLAN) and use the Rouge-L between the generated response and gold response as the regression target for the pair (prompt, generated response). To test the validity of Cappy, the authors evaluate Cappy in isolation, with no additional fine-tuning, on 11 held-out classification tasks from PromptSource. They show that Cappy consistently ranks the correct answer higher than many instruction-tuned LLMs, including OPT-IML-175B. Next, the authors evaluate Cappy on LLM augmentation using the BIG-Bench benchmark. For each task, Cappy is first fine-tuned, then used as a scoring function for multiple generations from a frozen FLAN-T5 LLM. Cappy consistently outperforms the proposed sampling methods, most notably improving upon the LLMs own self-scoring and In-Context Learning. Moreover, the authors show that Cappy can improve upon a FLAN-T5 model’s performance even when the LLM parameters are fine-tuned as well. Finally, the authors present an ablation study of their proposed training strategy for Cappy, showing that both the overall pre-training strategy and using Rouge-L regression targets from LLM generations are important to Cappy’s success.

Strengths

- Cappy represents a light-weight option to adapt LLMs to a given target task which requires no updating of parameters to the LLM nor any back-propagation through the LLM whatsoever (e.g. this is not true for adapter layers or prompt tuning). - Moreover, even without adapting to a given task, Cappy presents some benefits over generic LLM sampling methods for task-specific predictions. - Additionally, Cappy can ostensibly be added on top of any LLM without additional training (i.e. train once, use anywhere). - Cappy is a very simple idea, and is easy to understand, yet the usage of a scorer in this capacity, as well as the training and data augmentation scheme, is novel to the best of my knowledge. - The ablation study is helpful in demonstrating the benefits of the proposed pre-training and data-augmentation scheme.

Weaknesses

- Cappy is not compared to other parameter-efficient adaptation methods. While there is some justification given, because other methods require storing forward activations and back-propagating through LLM parameters, it nevertheless makes judging the effectiveness of Cappy difficult. - Moreover, Cappy’s improvements over a frozen LLM are comparatively small, e.g. while Cappy improves Flan-T5 Large performance by ~7%, fine-tuning alone increases performance by ~22%. It’s therefore unclear how significant Cappy’s benefit is without comparison to other adaptation methods. - In the zero-shot experiments (4.1) it is claimed that the improved performance of Cappy over much larger LLMs can be attributed to Cappy’s scoring-based learning strategy. However, T0 outperforms Cappy and T0 is significantly smaller than the OPT models; moreover, T0 is also trained on PromptSource only, similar to Cappy, and unlike the other LLMs. Thus, it seems that at least some of Cappy’s performance benefits compared to other LLMs may arise from domain mismatch between training and test domains, rather than the proposed scoring function. - While Cappy does only require a small number of parameters to train, inference requires a number of decoder forward passes to generate multiple candidate responses from an LLM (it's not clear how Cappy scales with the number of samples provided, also); to my knowledge, other adaptation methods such as Adapter Layers do not have the same limitation. So while it saves computation on one end, it does slightly increase computation on another end, and exploring this trade-off would be useful.

Questions

- I think it would be really helpful to understand how Cappy compares to other parameter efficient adaptation methods. Do you have any experiments that indicate that Cappy is e.g. comparable to prompt-tuning in terms of it's benefits? - Can you provide an analysis of how much additional computation is required to sample many generations for Cappy to score? - Also, do you know how Cappy's performance scales with the number of generated samples? e.g. does it's performance flatten out after only 4 samples, or can it's performance increase significantly if several generations are considered?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Yes.

Summary

In this paper, the authors tackle the challenge of computational requirements and memory constraints in fine-tuning Large Language Models (LLMs) by introducing an innovative approach that enhances LLM performance without the need for backpropagation through the LLM or access to its parameters. They propose Cappy, a pre-trained scorer that evaluates texts generated by LLMs based on downstream instructions and assigns them a score ranging from 0 to 1. To create Cappy, the authors curate a diverse collection of 39 samples from the PromptSource dataset, including good (ground truth), bad (random), and intermediate examples. The intermediate examples are generated using top-k and top-p sampling from BART0 and T0-3B models. Cappy assigns a score to each example based on the ROUGE-L metric compared to the ground truth response. This dataset is used to create a regression task and train a RoBERTa model that serves as the foundation for Cappy. The authors demonstrate improved accuracy across 11 classification held-out tasks from PromptSource compared to OPT, OPT-IML, T0, and BART0. They also showcase Cappy's effectiveness as an LLM booster for downstream adaptation on the BIG-Bench generative tasks, outperforming other selection strategies and achieving higher ROUGE-L scores on both frozen and fine-tuned FLAN-T5 models. Additionally, the paper discusses scenarios where Cappy performs less favorably than the "self-scoring" strategy on certain BIG-Bench tasks, suggesting that the lack of "memory" may contribute to this performance difference. Furthermore, the authors conduct an ablation study, revealing that data augmentation using LLMs is more crucial for improved performance than pre-training Cappy, although pre-training still offers benefits. Overall, the paper presents an innovative approach, Cappy, for boosting LLM performance without backpropagation or parameter access. The experiments showcase its superiority in classification tasks and downstream adaptation, while highlighting the importance of data augmentation and the potential limitations related to memory.

Strengths

- The idea of an auxiliary performance booster is an intriguing concept. The demonstrated improvements over the selected baselines highlight the effectiveness of this approach in enhancing language model performance. - The utilization of LLM generations for data augmentation and the regression-based evaluation using ROUGE-L scores is a novel and innovative methodology employed in this work. - Cappy introduces a unique integration of samples from LLMs and ground-truth information, leveraging correctness scores to enhance the quality of text generations. This ability to distinguish between good and poor responses is a novel idea that has the potential to enhance the performance of all language models. - Despite having significantly fewer parameters compared to large language models used in zero-shot baselines, pre-trained Cappy achieves comparable performance. This contribution is valuable as it demonstrates that Cappy can achieve performance on par with larger models, offering a more efficient and resource-friendly solution. - The versatility of Cappy is showcased by its potential for downstream adaptation and further fine-tuning to improve task-specific performance. This adaptability highlights the broad applicability and utility of Cappy as a solution in various contexts.

Weaknesses

- The performance of Cappy, trained on BART0 and T0 generations, does not consistently outperform T0 and OPT-IML. While the zero-shot performance is better than BART0 (a relatively smaller model), it falls short compared to T0 in most cases. The evidence supporting the claim of boosting language model performance is not sufficiently strong, and the improvements are not consistently observed. - The training of Cappy to optimize ROUGE-L scores against ground truth data raises concerns about the choice of this metric. ROUGE-L may not necessarily be the most optimal metric for evaluation, as alternative responses with low ROUGE-L scores may be valid and informative. Including additional metrics for evaluation would provide a more comprehensive understanding of the overall increase in performance. - The discussion on the tasks in BIG-Bench where Cappy performs worse than the "self-scoring" strategy is insufficient. Given the expectations for Cappy to perform at least as well, if not better, further analysis and exploration of these instances would strengthen the paper and provide a clearer understanding of the limitations of Cappy's performance.

Questions

- Could you provide an example illustrating how the limitation of instruction lengths affects in-context learning? Adding a specific example would enhance the understanding of this limitation and its implications. - In the context of downstream adaptation, it is not entirely clear whether the same LLM that is being augmented is used to generate the synthetic data. Clarifying this aspect in the paper would help readers better comprehend the process of downstream adaptation and the relationship between the augmented data and the LLM. - The reasoning behind the requirement for more "memory" in tasks like "sufficient_information" is unclear. It would be beneficial to provide a more detailed explanation either in the paper or in the appendix to shed light on the specific aspects that require increased memory in such tasks. - The results show that removing the Cappy pre-training step does not significantly impact the downstream score on BIG-Bench, implying that fine-tuning with augmented data from LLMs is sufficient. Have you explored the performance of RoBERTa initialization with data augmentation using LLMs? It would be insightful to investigate and report the results of this comparison to further validate the efficacy of the proposed approach. Minor Issues: - Line 186: PromprSource -> PromptSource - Line 285: 360 -> 360 M - Line 222: freezed -> frozen - This issue exists at several places in the paper.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors thoroughly discuss various limitations, including mathematical complexities, reliance on supervised datasets, absence of multi-lingual extensions, and other important concerns. However, it would be beneficial to further address limitations related to the reliance on generations from existing LLMs and potential biases that may arise as a result. Additionally, the use of ROUGE-L as the sole metric for regression and evaluation may not provide a comprehensive and accurate portrayal of the overall performance. It would be valuable to acknowledge this limitation and discuss the potential impact on the interpretation and generalization of the results.

Re: Rebuttal

Hi, thank you for the detailed response! I agree that the collection/generation of the dataset for training the cappy model is novel and interesting, and the way cappy is applied is simple. Additionally, thank you for the clarification with the contrastive data, that makes sense, and thank you for the rouge-l data. It’s interesting it's mostly centred around .2, which suggests lots of generations with low overlap - it makes me wonder if better balancing the distribution might improve results somewhat (but this is beyond the scope of this paper). However, I disagree that the methods in [1,8] are significantly different or fancier than what is proposed here. As stated on page 24 of [1], their best-of-n procedure is to “Sample N summaries from a supervised baseline at temperature 0.7, score them with a reward model, and take the summary with the highest score”. This is very similar to Cappy (argmax based on an answer rating module), but using slightly different decoding strategies, and using a reward model instead of Cappy. There are publicly available datasets for training reward models (e.g. https://huggingface.co/datasets/stanfordnlp/SHP, https://huggingface.co/datasets/Anthropic/hh-rlhf), and I think it is reasonable to compare a model trained using the Cappy data to models trained using these datasets - my guess is cappy would do better since it is focussed on correctness instead of helpfulness/harmfulness, but since the method is so close to prior best-of-n work, I think it is a necessary comparison. In my personal opinion, this is why I lean still reject and am not updating my score, but if other reviewers and AC are satisfied the method is novel enough, and the comparisons are rigorous enough, I think the other aspects of the work are solid and the empirical results certainly still useful for the community.