Time-Reversal Provides Unsupervised Feedback to LLMs

Summary

The paper proposes a class of LLMs called Time Reversed Language Models, which are simply pretrained on an unlabeled corpus in reverse order, and finetuned on instruction-tuned datasets accordingly. The model is used to provide feedback to LM generations in four different tasks. For general question answering, it serves to rerank N generated answers by scoring the probability of the question given the answers. For citation attribution, it scores a summarizing sentence given a cancidate text passage from the source article. For document retrieval, it scores the query given a retrieved candidate document. Finally, the model can be used to defend against jailbreak attacks by using the model to generate novel prompts given the response to the actual response. If the novel prompts exceed a threshold of rejection, the original prompt is rejected as well.

Strengths

* The proposed methods are simple yet effective ideas of how to use a reverse LM for various tasks. * While the idea to train a reverse LM is not novel per se, its applications are. * The baselines are well chosen to demonstrate the benefits of the model. The claims are well-supported. * On most tasks, the proposed method is very effective. * The discussed methods are of great significance to the NeurIPS community.

Weaknesses

* Even though the method is quite simple and easy to understand, I think the clarity of the paper could be better in some parts. For example, the purpose of the different parts/functions in Table 1 is not easy to understand without the context of the tasks that they are applied to. I think the paper could benefit from a "Problem Statement" section that introduces the different use cases from the tasks in Section 5 (e.g. reranking, retrieval, etc.) before presenting the model in Section 3.

Questions

Suggestions: * Section 6 is probably meant to be a subsection of Section 5. * Results are presented as "our method performs better by x%", although what is meant is the absolute improvement in points on the scoring metric. E.g., in line 304, the improvement of your best variant over the Forward Baseline is not 44.19% but 44.19 points, which really is more than 1200% better than the Forward Baseline. * I find Section 4 to be the least helpful part of the paper. As is acknowledged in the Limitations section, the assumptions are quite strong. I'd rather move it into the appendix in favor of using the additional space to enhance clarity as explained under "Weaknesses". However, I wouldn't object if the authors insist on keeping it the way it is.

Rating

8

Confidence

3

Soundness

4

Presentation

3

Contribution

4

Limitations

Limitations are appropriately addressed in a separate "Limitations" section.

Summary

This paper introduces Time Reversed Language Models (TRLMs), which operate in the response-to-query direction for scoring and generation. The key contribution is demonstrating that TRLMs can provide effective unsupervised feedback to improve language model performance on various tasks. Specifically, the authors show that TRLM-based reranking improves performance on the AlpacaEval leaderboard, citation attribution, and document retrieval tasks. They also demonstrate that TRLM's reverse generation capability can augment safety filters to reduce false negative rates in detecting toxic queries. The paper provides theoretical insights into why reverse scoring can be beneficial and empirically demonstrates the effectiveness of TRLMs across multiple applications, highlighting the importance of the response-to-query direction in language modeling.

Strengths

1. Novel approach: The paper introduces Time Reversed Language Models (TRLMs), which operate in the response -> query direction for scoring and generation. This is an innovative approach to leveraging unsupervised feedback in language models. 2. Theoretical foundation: The authors provide formal results and a bipartite graph model to explain how TRLM-based alignment can help mitigate certain forms of hallucination. 3. Comprehensive experiments: The paper demonstrates the effectiveness of TRLMs across multiple tasks: - Best-of-N reranking on the AlpacaEval leaderboard - Citation attribution - Document retrieval - Defending against jailbreak attacks 4. Significant improvements: TRLMs show notable gains over baselines across tasks, e.g.: - 5% improvement in length-controlled win rates on AlpacaEval - 44% improvement in citation attribution accuracy - 44% improvement in NDCG@10 for medical information retrieval 5. Versatility: The authors show TRLMs can be used for both scoring and generation tasks, demonstrating their flexibility. 6. Safety applications: The paper explores using TRLMs to augment safety filters against toxic queries, showing potential for improving AI safety. 7. Broad applicability: The techniques are shown to work across different model families (e.g., PALM2-Otter, Gemini-Pro, Mixtral), suggesting wide applicability.

Weaknesses

1. Assumptions: The authors acknowledge that the assumptions made for their theoretical results in Section 4 are stylized and may not hold true in practice. The hallucination model used is quite simplistic. 2. Need for more extensive testing of the defense strategy: While the proposed defense strategy for amplifying safety filters shows promise, the authors note it needs to be tested on a larger and more diverse set of questions to determine its broader applicability. 3. Computational costs: The paper does not thoroughly discuss the computational overhead of using TRLM for scoring and reranking, which could be significant for large-scale applications.

Questions

1. The paper focuses mainly on English language tasks. How well might TRLM generalize to other languages or multilingual settings? 2. The paper shows TRLM can help with safety filtering, but are there any potential risks or downsides to using this approach for content moderation? 3. How sensitive are the TRLM results to the specific prompts used for scoring and generation? Did the authors explore prompt engineering to optimize performance?

Rating

7

Confidence

1

Soundness

3

Presentation

3

Contribution

3

Limitations

Two suggestions for improvements: 1. More discussion of potential biases or fairness concerns in the TRLM approach would be valuable. 2. They authors could elaborate on privacy implications of using reverse query generation for safety filtering.

Summary

The paper explores the utilization of reverse/backward-trained causal LLM. This LLM can be used to score responses based on the probability of generating the queries given the scores (which can be combined with the probability of generating outputs from input). Then, they can be used for re-ranking. In general, they can be used to score sentence pairs - which can be used for citation attribution. The authors also consider the use of reverse LLM for toxicity filters. Given that standard output filters tend to be too aggressive in rejecting (high false negative), the authors found a way to improve the scores by generating the input from the output using the backward model, and then using the input filter for toxicity filtration. Reasonable baselines are considered and compared (for example, using the plain forward model for generating the query from response in a format fashion using an appropriate prompt), or alternative scoring methods (like perplexity).

Strengths

1. Application potentials of modern backward LLMs are not as well explored, so it's a relatively unique direction. 2. There are some interesting ideas for using backward LLMs, like toxicity filter besides standard ideas like re-ranking. 3. The paper provides some decent theoretical motivations for informing the scoring mechanism with a backward model.

Weaknesses

1. The core idea is not particularly novel. There are also a few missed citations. [1,2,3] (There could be others I am missing. Would recommend doing a deeper literature search) similarly used backward-trained models (one of the references is dialoGPT, which falls under the causal language model family) and their scores for re-ranking. 2. It may be a bit hacky to circumnavigate the output filter by projecting the output into the input space. In principle, the output being toxic doesn't mean an appropriate input has to be. RIght now, the hacky approach still may working betters because the alternatives are limited, [1] A diversity-promoting objective function for neural conversation models. - Li et al. NAACL 2016. [2] DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation - Zhang et al. ACL Demos 2020 [3] Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization - Zhang et al. NeurIPS 2018

Questions

Minor: in Line 183, I assume P(A|Q) = 1/|N(Q)| should be the appropriate notation.not 1/N(Q). Line 185: "the idea; ground truth" - I believe you meant ideal.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Response

Thank you for the rebuttal. Based on the rebuttal, I increased the score to 7. I am overall convinced by your points about the application of the approach for toxicity filter as more than a hack, and so I take back weakness 2. Since you are sampling multiple queries and looking at the percentage of rejection, it should be relatively robust against stochasticity, and the assumption that the probability of the input query being negative is high is reasonable. Regarding the discussion of related works. I am not too keen on some of the phrasings. For example: > "DialoGPT proposed by [2] pretrained a "backward" model to score sources given targets -- **this is specific to conversational understanding. This is different from results on general instruction following benchmarks we present, which are significantly more general and open ended.** For very large language models and on public, challenging benchmarks like AlpacaEval that test their instruction following capabilities, our work shows that full reverse token pre-training and reverse scoring gets the best set of results in re-ranking which is novel." It's not clear to me that instruction following is necessarily more open-ended and general than conversational understanding. You can always set up a conversational context to follow instructions or vice versa. Many powerful LLMs are already set up with conversational style prompts in their final instruction-tuned models. I don't recall DialoGPT doing anything so "specific" that it applies exclusively to explicit conversational setups. While sure they didn't have as exhaustive as an exploration as yours specifically for backward scoring and the current datasets were not even available back them but the point was more about the novelty of the core method, not differences in experiments, datasets, or parameters (how large the language model is). Either way I am still giving a higher score, because the ideas are still not explored previously in more modern contexts and the paper has novel application ideas, and experiments with well-made baselines and variations.

Thanks for the rebuttal

I acknowledge the author's response and would like to increase the rating.