Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models

Summary

In this paper, they frame alignment as a decoding time problem, allowing the large language model to be frozen. They do this by parametrizing a reward function with the difference in the log-likelihood between small untuned and tuned language models. Interestingly, their approach does not require shared vocabulary and is applicable to black-box LLMs. They perform experiments on controlled-sentiment generation, summarization, and instruction following.

Strengths

- The paper is well written and organized. - The motivation is clear from the beginning. - The experimental setting is adequate and the results are good.

Weaknesses

Although there’s a related work section, I believe the comparison between the proposed method and other methods, e.g., the ones that rely on small language models to aid the alignment of LLMs, should be described in more detail. In particular, the novelty of the proposed method should be made more evident; for non-experts, I think it’s a bit difficult to understand if some parts are novel or if they already exist in previous work. I also think that some citations might be missing (see my questions below).

Questions

Questions: - Is it \pi_base in L181? - What about the vocabulary of the small tuned and tuned models? Does it need to be shared? - How does your work compare with that of Li et al. (ACL 2023)? What about Zhao et al. (2024)? Minor comments: - BoN is defined in L204 but it’s used before (e.g., L166). References: - Contrastive Decoding: Open-ended Text Generation as Optimization (Li et al., ACL 2023) - Weak-to-Strong Jailbreaking on Large Language Models (Zhao et al., 2024)

Rating

7

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Summary

The paper introduces the "weak-to-strong search" method for aligning a stronger large language models (LLMs) by leveraging two weaker LLMs during test-time without requiring fine-tuning of the large models. This approach aims to improve alignment by maximizing the log-likelihood difference between tuned and untuned small models while using the frozen large model for decoding. The method demonstrates effectiveness in various tasks, including sentiment-controlled generation, summarization, and instruction following. I generally think this is an inspiring paper towards acceptance. The reason I rate it a weak accept rather than accept is I personally think an ablation study of using the log-likelihood difference to conduct token/chunk-wise PPO is necessary. It can make the paper more complete but lacking it is kinda fine though.

Strengths

1. The weak-to-strong search method is novel and provides a computationally efficient way to align LLMs without fine-tuning. 2. The method is shown to work across different tasks and with both white-box and black-box models. 3. The method is well-grounded in theory, with detailed mathematical formulations and explanations. I really like the overall presentation of this paper. The reading flow is good; better than many other ones in this NeruIPS review cycle. :D

Weaknesses

1. Obtaining a tuned weaker model is still not a very trivial thing, and the performance of $\pi_\text{base}$ on downstream tasks could heavily depend on the tuned weaker model as well. There is not too much discussion regarding it. 2. As we can formulate a partial reward function using a tuned/untuned model pairs, apart from using it to guide decoding, a natural next step/ablation study is using it to conduct PPO. It is worth trying this experiment and compare the downstream performance as well as the overall cost.

Questions

N/A no questions here; the paper is presented very clearly.

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The paper includes a limitation section at the end.

Summary

The paper proposes a search/decoding method "weak-to-strong search" for improving LLM's performance at test time by leveraging the log likelihood from small language models. Specifically, it utilizes the log likelihood differences between tuned and untuned small models to guide the decoding of larger models. This approach avoids the computationally intensive process of fine-tuning large models by using smaller, more manageable models for alignment at inference time, guiding the large model to generate better responses. The paper demonstrates the effectiveness of this method across various tasks, including controlled-sentiment generation, summarization, and instruction-following tasks.

Strengths

* The paper introduces a novel approach to improve large model alignment at inference time, which reduces computational costs compared to fine-tuning methods. The idea of leveraging log likelihood differences between untuned and tuned small language models is quite interesting. * The method is tested across multiple tasks and has shown effectiveness. Results demonstrate improvements in multiple tasks. * Ablation studies offer further analysis and understanding of the hyperparameters and components of the method.

Weaknesses

* Although the method avoids the computational costs associated with fine-tuning, the paper does not explain clearly the overhead introduced during inference. How does weak-to-strong search impact memory usage and inference speed? * How does the proposed method compare to other inference-time techniques, such as in-context learning (using few-shot examples) and prompting methods like CoT? * The method appears to be broadly applicable and not strictly limited to a weak-to-strong approach. For smaller models, such as Llama3-8B as used in the experiments, why would this method be preferred over techniques like LoRA fine-tuning, which can be performed at a low cost and yield better performance? Additionally, if the target model has already been fine-tuned, can the proposed method still enhance its performance? * (minor) It would be interesting to explore how weak-to-strong search could enhance a model's reasoning abilities, for example on math datasets like GSM8K.

Questions

See weaknesses above.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, the authors discussed limitations in the paper.

Summary

This paper addresses the alignment of large language models without the need for fine-tuning. It conceptualizes the alignment process as a search problem, leveraging the log-likelihood difference between small tuned and untuned language models as both a reward and a critic. By transforming a sparse preference reward into a per-token dense reward, the method achieves weak-to-strong generalization. The effectiveness of the proposed approach is empirically validated through controlled sentiment generation, summarization, and an instruction-following benchmark.

Strengths

1. The paper is well-written, and the empirical results are promising. 2. The method is well-motivated and theoretically sound. 3. The proposed approach provides token-level guidance instead of sparse sequence-level rewards, enhancing precision and control.

Weaknesses

1. The paper does not include comparisons with existing decoding-based alignment baselines [1,2]. 2. The token-based MDP formulation is largely derived from existing work [3]. The main innovation of this paper lies in using the log-likelihood difference between two small models for guidance based on the existing formulation. [1] ARGS: Alignment as Reward-Guided Search, ICLR 2024 [2] Controlled Decoding from Language Models, ICML 2024 [3] From r to Q*: Your Language Model is Secretly a Q-Function.

Questions

What is the difference between the formulation in section 4.1 and the results in [3]?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The paper has discussed limitations.

Thank you for your rebuttal

Thanks for your response. I appreciate the rebuttal and have read other reviews. I maintain my score as 6 and will engage in further discussion with other reviewers and AC if needed.