On Giant's Shoulders: Effortless Weak to Strong by Dynamic Logits Fusion

Summary

The article discusses existing weak-to-strong methods, noting that current approaches typically use a static knowledge transfer ratio and a single small model to convey complex knowledge, which results in suboptimal performance. Consequently, the article proposes a dynamic logit fusion method that employs a series of task-specific small models and adaptively allocates weights among these models at each decoding step. The weights are learned by optimizing a problem constrained by Kullback-Leibler divergence. The article conducts experiments on various benchmarks, including both multi-task and single-task scenarios.

Strengths

1. The article reevaluates existing logit arithmetic methods, highlighting the significant impact of fusion weights and the limitations of a single small model on test performance. 2. By using constrained optimization, the article autonomously learns fusion weights, thereby approximating the computationally intensive results of fine-tuning large foundational models. 3. Experiments were conducted to validate the proposed method, demonstrating notable improvements in performance, generalization capability, and robustness.

Weaknesses

1. In Section 3.2, the proposed method is based on an assumption that lacks clear supporting evidence. Does this somewhat undermine the theoretical foundation of the algorithm? 2. In the multi-task experiments, the results for CNN/DM do not demonstrate the method's superiority. Additionally, in the experiments on unseen tasks, the method does not show significant improvement. 3. In the experiments, all evaluations were conducted in a 0-shot setting. How would the evaluation results change if a 5-shot experiment were conducted?

Questions

1. In the multi-task setting, which 7B model is used as the expert to implement your algorithm on the 13B model? I am very confused. If you are using different sets of experts to operate on the 13B model, isn't this weak-to-strong? Because the parameters of multiple 7B models exceed 13B. 2. There seems to be a typo in formula (9) in Appendix B, it seems to be $\propto$ rather than $=$.

Rating

4

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors have adequately addressed the limitations.

Summary

They tackle the problem of merging the logits from multiple models. To do so, they propose an objective that minimizes the squared loss of the KL between the two pairs of (student, teacher) models. This is solved via a random search.

Strengths

- Paper well written and easy to follow - Nice ablations on alpha and efficiency - Works well in single task scenarios

Weaknesses

- Using the squared error between two KL’s is not theoretically motivated (at least that I am aware of) - More description of the optimization method in main text since it is a big part of the method - Missing baseline in multi-task tuning setup

Questions

- Where is the baseline proxy tuning for multi-task tuning in Table 2? - From algorithm 1 in Appendix C, the optimization is done just via random search? Guessing values and storing the best? I couldn’t find the method to optimize the objective mentioned in the main paper. It should be more clearly stated in the main paper, and how it handles multitask setup. - For the efficiency analysis, the BV can be done in 1 forward pass all in parallel, but the n parameter searches require 20 forward passes. It really is n times more compute, which is not minimal. For example, decoding 100 tokens vs decoding 1 token would be a constant in the Big O, but they are different efficiency wise. - In eq (2), is the second term (multiplied by alpha) not normalized but in logit space? - In fig 4, are there some special about the tokens being generated at the timesteps where $alpha$ is high?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

NA

Summary

The paper studies the problem of adapting large general language models via smaller expert language models fine-tuned on specific tasks. Prior work proposed the idea of mixing logits between a large model and the differencei in logits pre- and post- finetuning of a small model. The authors take this idea a step further and compute the mixing weights adaptively per-token, leading to better results.

Strengths

1. The method is very simple: the authors match tune the weights to match the KL divergence between the small model before and after fine-tuning, for each token. 2. The experiments are comprehensive with 5 tasks and 2 small models (1.1B and 7B). The authors also consider both single-task and multi-task scenarios. 3. The results are good across all tasks, the proposed method outperforms proxy-tuning as well as full fine-tuning on the smaller model predictions, and recovers a large fraction of the ceiling performance achieved by directly finetuning the large model on ground truth. 4. There are several ablations and understanding experiments in Section 5.

Weaknesses

1. It is not intuitively obvious to me why matching the KL divergence is the right objective. Could the authors please provide some intuition? I imagine it is something like this: when the small model updates significantly for some token, we want the large model to also udpate significantly. That seems reasonable, but probably doesn't always work well: if the small model is unaware of some fact or makes an arithmetic mistake, it may need to update significantly on the corresponding tokens, while a large model does not need to update. 2. The presentation is not always very clear. For example, in Eq. (5) it is not clear to me what the authors mean by the joint distribution of $Q_1, Q_2, \ldots, Q_T$. How can we compute KL between this joint and $Q$? 3. As the authors mention, the proposed method is 2.5 times slower at inference time compared to standard sampling from the same model.

Questions

1.1 and 1.2: See weaknesses 1 and 2. 2. What exactly do you mean by supervised instruction tuning: what are the smaller models fine-tuned on? Are these chain-of-thoughts for solving the task, e.g. GSM8k? Where do they come from? 3. In Figure 3, qualitatively, what do the tokens (decoding steps) where we set the weight to the lower bound correspond to, and same for the upper bound? Are they qualitatively different? 4. In Figure 3, why is the lower bound 0.8 and the upper bound 1.5? Are these tunable parameters? 0.8 seems quite high, shouldn't we want to set the lower bound to 0?

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Limitations are adequately discussed.

Summary

This paper focuses on the weak-to-strong generalization paradigm where the goal is to transfer knowledge from a small language model to larger one. The method they study is the one proposed by Mitchell et al. [1]: they use log probability algebra to combine the logits of the large model, the ones of a small model and the ones of a small model that has been finetuned. This combination involves a parameter $\alpha$ that controls the contribution of the small model. The main contributions of this paper is to point to the limitations of using a static $\alpha$ and to propose a method to adaptively learn such $alpha$. Their method consists in optimizing an objective based on the KL divergence and they show that their approach is consistently better than using a static alpha across a wide range of downstream tasks. [1] Mitchell, Eric, et al. "An emulator for fine-tuning large language models using small language models." arXiv preprint arXiv:2310.12962 (2023).

Strengths

I find the paper is well written and the methodology well presented. The authors did a great job at presenting the problem, the limitations of the current methods to solve the problem and their method. They also did a good job at presenting their experiments.

Weaknesses

Overall, my main concern is that I find the contribution limited and I have some doubts about the method. Here is a detailed list of my concerns: - **Computational feasability**: I think that the authors should be more transparent in the computational cost of their procedure. Solving the optimizaiton problem at every decoding step may be very expensive and it is not clear to me that when one needs to do many generations, a finetuned model with LoRA on the large model is cheaper than the procedure proposed by the authors. Also, is it important to update alpha at each decoding step? Can't one get a more efficient procedure by updating it only every 100 tokens or so? - **Not enough clear to me that the method does much better than the static $\alpha$**: when I see the barplot of figure 2, it seems like $\alpha=1$ is a bit below than the learnt $\alpha$ but the gap is not huge. - **Theoretical justification for the optimization problem?**: So if I understand correctly, the authors objective function is to say "I want the distance between the predictions of $\tilde{P}$ and the large model is the same as the distance between the predictions of the finetuned small model and the small model". This looks like a reasonable belief. However, it would have been nice to have a theoretical justification. For instance, when you do RL finetuning, if you solve the problem exactly, the distribution you end up generating from ends up being p_bayes(generation) * exp(reward model you are training on) (usually done with ppo, dpo). When you take the logs, you get log(p) = log(p_bayes) + reward model. Then you can estimate the reward model by taking the difference of logits for any model scale and in this case the intuition of the authors make sense to me and it is principled. However, in the standard finetuning case, when the authors apply this intuition at the level of tokens, it is not clear to me why it should work. - **Scaling experiments for studying the approach**: I know that the Llama suite starts at 7B but it would have been nice to study the behavior of the method with smaller models than 7b. Understanding how robust the method is by varying the gap between the weak and strong models is important. There are maybe chances that the learnt $\alpha$ approach shows bigger gaps with respect to the static $\alpha$ when the gap between the weak and strong models is large.

Questions

I would appreciate that the authors address the concerns I have regarding the theoretical justification of their optimization problem, the computational cost of their approach.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

I think that the authors didn't clearly state the limitiations of their approach, which is regretful.