Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models

Summary

This paper proposes Reference Trustable Decoding (RTD), a new paradigm that allows models to adapt to new tasks without fine-tuning and has lower inference costs compared to in-context learning (ICL). RTD uses the last-hidden states of the input sample to retrieve similar samples in the training examples and optimizes the LLM’s final vocabulary distribution using these examples’ last-hidden states. Experimental results on various LLMs show RTD can effectively adapt LLM and it can be used together with ICL method to further boost task performance.

Strengths

1. The proposed method is well-motivated, addressing the weaknesses of ICL and fine-tuning in model adaptation. The paper is also clearly written, with Figure 2 providing a clear comparison of RTD and baseline methods. 2. The experimental results on multiple task setups showcase the effectiveness of the RTD in adapting the model. Also, the proposed method can be used together with traditional methods to further boost performance. 3. Compared to RAG or ICL, RTD selects relevant examples based on the similarity in the final hidden state space. Many existing techniques for vector database can be used to boost the efficiency of RTD.

Weaknesses

1. The description of using RTD in language generation task is unclear. When generating a sequence of multiple tokens in an autoregressive way, how does the retrieved sample adapt the vocabulary distribution after the first token? Also, it is unclear what the constructed “benchmark based on RAG and Open Book Question Answering” in Section 4.2 is. 2. The performance gain of RTD seems to be unstable as Table 2 shows on some datasets, RTD is much worse than 5-shot ICL. The paper lacks discussion on what tasks is RTD suitable to use. 3. The paper should have a part clearly describe the baseline methods in comparison. How are the 5 examples selected in “5-shot ICL”? I think this is quite important, as some previous works use similarity in the sentence embedding space to select in-context examples to adapt the model via ICL. Given the proposed method mainly leverage the last hidden state space, more careful comparison and discussion can help understand whether it’s more helpful to leverage the last hidden state space.

Questions

See my questions in “Weaknesses”.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes. It is discussed in "Limitations & Future Work".

Summary

The paper introduces Reference Trustable Decoding (RTD), a novel framework that allows large language models to adapt to new tasks without the need for fine-tuning. RTD constructs a reference datastore from training examples and optimizes the LLM’s final vocabulary distribution by selecting suitable references based on the input. This results in more reliable responses and low-cost adaptation to downstream tasks. The paper includes experimental evaluations demonstrating RTD's effectiveness across various benchmarks and tasks, highlighting its potential as a new paradigm for augmenting LLMs.

Strengths

1. The paper presents a unique and training-free method for augmenting LLMs, which is a significant contribution to the field. 2. The methodology is clearly explained, and the figures provided help in visualizing the RTD process. 3. Comprehensive experimental evaluations are conducted, demonstrating the effectiveness of RTD across different tasks and benchmarks. 4. The paper discusses the orthogonality of RTD with traditional methods, indicating its potential for combined usage and enhanced performance. 5. The authors provide a thorough analysis of the hyper-parameters involved in RTD and their impact on performance.

Weaknesses

1. While the paper discusses the efficiency of RTD, it does not provide a direct comparison with other methods in terms of computational resources or scalability. 2. The paper could benefit from a more detailed discussion on the potential applications of RTD in real-world scenarios. 3. The experimental section, although comprehensive, might have benefited from additional benchmarks, particularly those that test the generative capabilities of LLMs further. 4. The paper mentions the inability to compress large reference datastores as a challenge; however, it does not provide insights or potential solutions to this issue.

Questions

1. How does RTD compare with other state-of-the-art methods in terms of computational efficiency and scalability, especially for larger datasets or more complex tasks? 2. Can the authors provide more details on how RTD might be integrated into existing systems or workflows where LLMs are already deployed? 3. The paper mentions the potential for RTD to be combined with other methods like ICL and fine-tuning. Are there specific strategies or best practices for such integration?

Rating

4

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

1. The paper acknowledges the challenge of compressing large reference datastores but does not offer solutions or workarounds for this limitation. 2. The scope of the experiments could be expanded to include a broader range of tasks, especially those that are more generative in nature. 3. While the paper discusses the performance of RTD in a controlled setting, it does not address how well these results might generalize to other domains or tasks outside the tested benchmarks. 4. The paper does not provide a detailed discussion on the ethical considerations or societal impacts of deploying LLMs with enhanced capabilities through RTD, which could be important for responsible AI development.

Summary

Adapting large language models (LLMs) to specific tasks remains costly and complex. In-context learning (ICL) and parameter-efficient fine-tuning (PEFT) still suffer from inference latency and training costs, respectively. To address these problems, this paper proposes Reference Trustable Decoding (RTD), a training-free paradigm for better adapting base LLMs to downstream tasks. RTD is a retrieval-based framework that creates a reference datastore from task datasets, storing key-value pairs of hidden states and corresponding labels. During inference, RTD retrieves the most relevant references based on the hidden states of the input and then aggregates them with model logits to compute the final output distribution. Experimental results demonstrate that RTD effectively adapts mainstream LLMs for natural language understanding (NLU) and natural language generation (NLG) benchmarks, outperforming ICL approaches.

Strengths

1. **Efficient Adaptation:** The proposed RTD method integrates external knowledge and constraints to adjust outputs without increasing input length or using gradient descent. 2. **Orthogonality:** As a plug-and-play decoding paradigm, RTD can be flexibly integrated with existing techniques such as ICL and fine-tuning. 3. **Significance:** The paper demonstrates the excellent performance of the RTD method on multiple benchmarks, surpassing ICL methods and baselines. The RTD method is particularly meaningful in scenarios requiring rapid iteration of content or controllable generation.

Weaknesses

1. **Related Work:** As an RTD method based on retrieval, the experiment sections should discuss and compare it with similar studies such as kNN-LM [1] (which also is cited in the paper) and other retrieval-based decoding methods. 2. **Generalization Analysis:** The proposed method relies on the construction of a reference datastore, however, the discussion on out-of-domain inputs is still lacking. This poses a potential risk to the generalization of RTD method. 3. **Hyperparameter Challenges:** Ablation experiments show that the effectiveness of RTD relies on adjusting several hyperparameters, which may hinder the method's scalability. In practical use, applying the RTD method to new models and tasks may require additional hyperparameter searches on a held-out validation set. 4. **Trustable Decoding Association:** The discussion in the paper may not be strongly associated with "Trustable Decoding" [2]. Utilizing a reference datastore provides transparent reference data for base model decoding, achieving in-domain controllable generation. However, the core argument of the paper still emphasizes efficiently transferring base LLMs using training-free methods. 5. **Typos:** - Line 242: "Detailed" should be "detail." References: [1] Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., & Lewis, M. (2020). Generalization through Memorization: Nearest Neighbor Language Models. In International Conference on Learning Representations. [2] Liu, Y., Yao, Y., Ton, J. F., Zhang, X., Cheng, R. G. H., Klochkov, Y., ... & Li, H. (2023). Trustworthy LLMs: A survey and guideline for evaluating large language models' alignment. arXiv preprint arXiv:2308.05374.

Questions

In NLU tasks, why is the value of $\lambda$ set to 1? Under this setting, the distribution of the output tokens entirely comes from RTD, without integrating the outputs from the LM head itself. Lines 268-269 also mention that the value of $\lambda$ empirically ranges between 0.4-0.7. Setting it to 1 might require some justification or explanation in the paper.

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors have included Limitation section after conclusion.

Conclusion and General Response of Paper 9447

Dear PCs, SACs, ACs, and reviewers: In our paper, we introduce the Reference Trustable Decoding (RTD) method and its multi-head variants as an efficient and low-cost approach to augmenting the capabilities of Large Language Models (LLMs) for downstream tasks. By leveraging an additional Datastore, we achieve the effect of editing the *LM_Head* layer without requiring additional training. Since our focus is on the later layers of the model, we gain greater flexibility and interpretability. Moreover, we avoid reliance on gradient-based backpropagation training methods, achieving rapid performance improvements or knowledge editing while maintaining efficiency. Our method incurs minimal additional cost during inference and benefits from mature optimization techniques. Compared to the excessively long KV Cache produced by ICL (In-Context Learning), our approach is more efficient. Additionally, Datastore generation is on par with inference costs, avoiding the need to retain optimizer parameters and intermediate states as in fine-tuning methods. Furthermore, it involves less computation and faster speed. We evaluate RTD across various benchmarks, including both language understanding and language generation tasks, as well as different multilingual models. RTD consistently performs on par with other augmentation methods. We extend our heartfelt gratitude to all the contributors involved in reviewing this paper, including PCs, SACs, ACs, and all reviewers. Your time and effort are invaluable to us, and your insightful suggestions have significantly improved the quality of our work. In particular, we appreciate the feedback from reviewer *tphZ*, who raised questions about the applicability scope, prompting us to further evaluate the strengths and limitations of RTD, and the recognition of our work by the reviewer. We also acknowledge reviewer *wxwA* for highlighting issues related to generalization and trustable discussions in our original paper, which guided our paper’s optimization. Following discussions, the reviewer agreed to raise certain scores. Additionally, we thank reviewer *vZ3x* for their extensive inquiries; our timely responses facilitated valuable revisions to our article. Finally, if the reviewers could reconsider the scores by taking into account the additional content provided during the rebuttal and discussion stages, as well as our commitment to incorporating suggested improvements in the next revision, it would be a great honor for us. Once again, we would like to express our gratitude to everyone who contributed to our paper. Your dedication and insights have been immeasurable for us. Best Regards, Paper 9447 Authors.