Tuning Language Models by Proxy

Summary

This paper introduces proxy-tuning, a resource-efficient method that adapts LLMs without needing access to their parameters. By tuning a smaller LM and applying the predictive differences to a larger, untuned model, proxy-tuning effectively mimics full-scale tuning. Experiments show it can nearly match the performance of fully-tuned models in various benchmarks, including knowledge, reasoning, and safety, and even surpasses them in truthfulness on specific tests. The technique is also applicable to domain-specific adaptations and updating models with recent knowledge.

Rating

8

Confidence

4

Ethics flag

1

Reasons to accept

1. Research on how to customize and efficiently "fine-tune" LLMs is important. 2. The proposed proxy-tuning is innovative. 3. The paper is well-written, and its experiments are solid.

Reasons to reject

1. There have been some similar ideas to this paper’s idea, such as DEXPERT for detoxification and VDD [2] for reducing hallucinations or bias. It's unclear how much we can benefit from this paper’s idea. [1] Debiasing Multimodal Large Language Models. https://arxiv.org/pdf/2403.05262

Questions to authors

1. Why does the proposed method adjust the output probability of the larger model using the difference between the tuned small model and the untuned small model? Why not directly adjust the probability of the larger model based on the tuned model? 2. Does the proposed model adjust the output probability of the larger model step by step, or does it adjust only once after generating all content? 3. If closed-source LLMs do not provide output probabilities or logits, does it mean that the proposed method will not work? 4. In Table 1, could you provide examples of tuned LLAMA2-13B and compare them with cases of proxy-tuned LLAMA2-13B? 5. Could you clarify why, in Table 2, Directly Tuned 70B underperforms Directly Tuned 13B? Additionally, why does Proxy-Tuned 70B significantly outperform Directly Tuned 70B in the last column, yet underperforms Directly Tuned 70B in other cases?

Summary

The paper introduces a technique reminiscent of contrastive decoding to 'tune' a large LLM without gradients by shifting the logits of each next-token distribution using the difference between logits under two smaller models (one which is tuned for the relevant task and another which is not). Given iterative access to all relevant generators and next-token distributions, the technique is fairly straightforward to apply and the paper demonstrates its effectiveness across various benchmarks. Most experiments adapt Llama 13b or 70b using adapted and unadapted versions of Llama 7b. In one experiment, the paper demonstrates (in a rather limited scenario, where a single-token response is sufficient) that the technique can be used in a setting as closed-access as that of adapting ChatGPT (provided access to top-5 logits). The paper is reasonably clear, and complete (to the best of my ability to assess it).

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

A clearly written paper that presents a simple and effective decoding-time technique to adapt a large base LM without access to its internals (but requiring iterative access to its generation algorithm and requiring the logits that parameterise the next-token distribution at each step).

Reasons to reject

I can list a point for improvement, but I think it only requires clarification before a final revision (ie, it's not a reason to reject the current version). It's not too clear to me when this method would be needed. When do we have access to all that it requires but do not have the ability to tune the large model? The example of ChatGPT isn't a good one: while it allows a demonstration of the method, that demonstration is in a rather artificial and limiting single-token response setting. I think it's okay if the answer to this question can be seen as a bit disappointing, but I think this should be discussed clearly and openly in the paper.

Questions to authors

In one experiment reported in this paper, proxy tuning beats direct tuning in Truthful QA, this can be due to anything, something about the method, something about the data, something about this one experiment. I think the paper would be better without the following attempt at an "explanation" _"The improvement in truthfulness suggests that decoding-time algorithms may preserve knowledge better than direct finetuning, which sometimes hurts knowledge-intensive tasks"_. Besides being shallow, it doesn't explain, it makes further claims that you cannot test easily.

Summary

This paper tackles a problem setup that, given a small accessible LM and a large inaccessible LM, how we can close the performance gap between the small and large model. This setup is highly relevant to many researchers and practitioners who lack the resources to train the largest LMs, such as those with 70 billion parameters. The proposed approach is fairly simple: First, it fine-tunes a base model on the target task, then adds the logit offsets between the tuned and base models to the logits of the larger (untuned) model, followed by softmax. The adjusted distribution is then used for the final generation by the large model. The authors rigorously evaluate this approach on a range of target tasks, from instruction tuning to code adaptation. The results of instruction-tuning experiments strongly support the effectiveness of this approach, achieving the win rate of 88% on AlpacaFarm and 32% on GSM. Proxy tuning even outperforms the directly tuned model in terms of accuracy on TruthfulQA, which intensively requires world knowledge. These positive trends are consistent in the code adaptation experiments as well. Additionally, the authors demonstrate a use case of proxy tuning with proprietary LMs such as GPT-3.5, which boosts the performance on RealTimeQA.

Rating

8

Confidence

4

Ethics flag

1

Reasons to accept

- A simple yet effective approach for managing large language models (e.g., 70B-parameter models) in limited-resource settings. This approach opens up opportunities to researchers and practitioners who lack the resources to train the largest LMs. - The strong experimental results demonstrate the effectiveness of this approach. - This paper is very well-written, clearly describing the approach and its evaluation.

Reasons to reject

- Although this approach doesn’t require updating a large LM itself, it still requires training a small LM and performing forward computation for three times during inference time. This might be costly in practice. - This approach assumes a situation where a small accessible LM and a large inaccessible LM are available. Therefore, this approach can be used for LLM families that release different sizes and versions. Although this is typically not a problem in practice (i.e., model developers usually release different sizes and versions), one could argue that it is a strong assumption.

Questions to authors

- Have you observed any failure modes after applying proxy tuning? Particularly, it would be interesting to see losses where the base model gets right, but proxy tuning gets wrong. - Have you investigated different temperature values in softmax? Or, a vanilla softmax just works fine? I wonder if this is task dependent.

Summary

This paper proposes the proxy tuning method, which uses tuned and untuned small language models to steer the predicted logits of a larger LM at decoding time. The predicted logits are offset with the difference between the logits of tuned and untuned small LMs. The authors evaluate their proposed method of instruction-tuning, domain adaption and task-specific for the LLaMA-2 family and temporal adaptation for the proprietary model (GPT-3.5). The experiment results show that the proxy-tuned LMs significantly outperform the base models and approach the performance of the directly tuned models in some cases.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

- While the proposed method is simple, it is efficient and works well in multiple scenarios. - Well-designed experiments to show the benefits of the methods in common LLM usecases.

Reasons to reject

- It is intriguing to understand the underlying conditions and reasons for the method's effectiveness. The authors' analysis or hypothesis on this matter would undoubtedly enrich the paper. I speculate that, in addition to requiring the same vocabulary, it is crucial for the small and large LMs to be trained to approximate the same data distribution. This could explain why the proxy-tuned model cannot surpass the CodeLlama-7B directly-tuned model, as CodeLlama-7B has been adapted to a new data distribution (code data). This can also explain the limitations in the minor improvement of GPT-3.5 experiment.

Questions to authors

1. How do the predicted logits of proxy-tuned and directly-tuned models look like? Do the proxy-tuned logits have a similar pattern to the directly tuned model?

Response to Authors

Thanks for your detailed clarification. Although my concerns have been addressed, I will not increase my score as I have already assigned a very positive one. Best of luck!