InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models

Summary

The paper presents an innovative approach to identifying the "optimal" instruction for large language models with the aim of improving generative quality. This work falls under the burgeoning area of prompt search methods, which have gained significant attention recently, exemplified by methods such as APE, RLPrompt etc. Unlike conventional methods that optimize discrete instructions, the authors propose optimizing a low-dimensional "soft prompt" using dimensionality reduction. The optimized soft prompt is applied to an open-source Lifelong Learning Model (LLM) to generate instructions for a black-box LLM. The optimization process is iterative, involving zero-shot evaluation of the black-box LLM's performance, which is then used in a Bayesian optimization scheme to refine the soft prompts. This iterative process continues until convergence. Experimental results on the BBH benchmark show that the proposed method yields superior performance across all 32 tasks.

Strengths

- The proposed methodology is both innovative and well-explained, making a valuable contribution to the area of prompt optimization in large language models. - The empirical results are compelling, demonstrating superior performance across all 32 tasks on the BBH benchmark. - The use of Uniform as a comparative baseline effectively underscores the benefits of the proposed iterative Bayesian Optimization (BO) process.

Weaknesses

- The paper could benefit from a broader evaluation scope. Considering additional tasks such as reasoning QA GSM8K, machine translation, or summarization could provide a more comprehensive view of the method's effectiveness. - The inclusion of only two comparative baselines, APE and Uniform, limits the robustness of the evaluation. Expanding the set of comparative baselines could provide a more holistic understanding of the method's performance relative to existing work. For example, RLprompt and Autoprompt are also two good prompt search methods. - The paper presents a puzzling result related to APE's performance, which is reported to have only a 0.04 accuracy in Figure 1. Upon closer inspection, it becomes apparent that the original APE experiments were based on instructgpt, where the prediction probability could be obtained. While this paper employs the more powerful Turbo 3.5 API, which cannot access the prediction probability. Thus I think the comparison here is not fair enough, as APE is a much weaker version than the original paper. This discrepancy introduces confusion and could affect the perceived validity of the comparative results. This drawback again highlights a more fair comparison is required, e.g., other prompt search baselines are needed. Also, what about the comparison with zero-shot COT, 'Please Think step by step'? - There is ambiguity in the claim about the method being applicable for zero-shot evaluation. While it's true that the proposed method employs a black-box LLM API for zero-shot generation, the Bayesian Optimization (BO) process requires labeled data. This seems to contradict the zero-shot claim and may constitute an overstatement.

Questions

- I wonder why the results of APE are so weak in Figure 1.

Rating

6: marginally above the acceptance threshold

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

4 excellent

Contribution

2 fair

Summary

This paper proposes a method to optimize the instructions for black-box large language models. The proposed method uses an open-source LLM to convert a soft prompt to an instruction, and then uses the instruction as input to the black-box LLM. Bayesian optimization is then used to optimize the soft prompt, which can iteratively propose new soft prompts and instructions to be evaluated by the black-box LLM.

Strengths

- The proposed method of using another open-source LLM to help convert a soft prompt to an instruction and then using the instruction as input to the black-box LLM is an interesting and intuitive idea. - The graphical illustrations in Figure 2 and 3 are nice and helpful for understanding. - The results in Figure 4 indeed show that the proposed method improve over APE and Uniform.

Weaknesses

- One overall observation from the experimental results which concerns me is that it seems that APE does not consistently perform better than Uniform? Both Figure 4 and Figure 1 seem to suggest this, for example, in Figure 1, the improvement over APE seems to be larger than over Uniform. This is an unexpected observation and I think should be explained, because it may suggest that performances of APE might be underestimated in the experiments here. - I have some questions and concerns about the instruction-coupled kernel. First of all, it seems that to calculate this kernel between a pair of input soft prompts, you need to have the evaluated scores for both soft prompts (correct me if I'm wrong)? If this is the case, then when you calculate the vector $\boldsymbol{k}$ in equations 4 and 5, this instruction-coupled kernel cannot be used to calculate these kernel values and therefore these kernel values will simply use the normal squared exponential or matern kernel? In this case, I wonder how much this instruction-coupled kernel actually helps the performance of the Bayesian optimization, because the vector $\boldsymbol{k}$, which directly measures the distance between a new soft prompt and other previously evaluated soft prompts and therefore has a huge influence on the uncertainty measure, cannot make use of it. I see that you have an ablation study in Table 4 to show the effect of using the instruction-coupled kernel, but why did you only show the comparison for a small number of selected tasks? I think to see whether this kernel is actually useful, it's important to fairly run this ablation study in all tasks and make an overall comparison. - About the ablation study (Section 4.3), it looks like the scores "w/o Manual" is in general better than "Manual"? This is also puzzling because it implies that the meta-prompt used by APE may not be useful... - The proposed method InstructZero seems to only optimize the zero-shot performance of the instructions instead of few-shot performance. However, since you already have access to these input-output exemplars which are used as input to the open-source LLM, why don't you also use them as input to the black-box LLM to improve the performance? So this may bring into question how practical the experiments are. - (minor) Equations 4 and 5, it seems that the matrix $K$ is not explained.

Questions

My questions are listed under "weaknesses" above.

Rating

3: reject, not good enough

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Summary

This paper proposes to use Bayesian optimization to learn an instruction with an open-source LLM so that the instruction improves the zero-shot results of a black-box LLM. Since instructions are discrete, this work instead iteratively learns a small soft prompt which then gets decoded as an instruction. Each updated instruction is evaluated on the black-box LLM whose training accuracy is used to find a better soft prompt.

Strengths

This is an interesting direction toward automating prompt engineering for API models, and shows strong results.

Weaknesses

It would be equally interesting to see qualitative analysis of the errors and various failures modes by the method and the different components used for optimization (e.g. open-source/black-box LLMs).

Questions

Some of the similarity metrics are chosen because black-box models don't necessarily return the log-probs. An ablation could have been run where an open-source model is used for both instruction proposal and loss evaluation. Then, we have access to log-probs and/or gradient and will have a better understanding of how much performance we are losing. Could be interesting, not saying this should have been run.

Rating

8: accept, good paper

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

General Response

We appreciate all reviewers for providing useful feedback! We hope our responses address your concerns. Please feel free to raise further concerns and we are open to discuss. Reviewers #FwK3 and #4arY both point out that the performances of APE might be underestimated in our experiments. We would like to clarify that the original APE can access 100 demonstrations~(input-output pairs) for instruction optimization. In every iteration of APE, they randomly draw 5 new demos from a hold-out set of 100 demos. Instead, our hold-out set contains only 5 demos and InstructZero does not have access to more new demos---this is more practical and also more challenging. In our experiments, for a fair comparison, both APE and InstructZero have access to the same hold-out set for instruction induction and use GPT-3.5-turbo as the API LLMs. In addition to the reported 5-demo experiments, we expand InstructZero's hold-out set of demos to 10 demos and compare it with the original APE using a hold-out set of 100-demos. We keep the API LLM the same as the one used in our original manuscript, i.e., ChatGPT. The results are shown as follows: | Dataset | APE (5 demos) |APE(100 demos) | Ours (5 demos) | Ours(10 demos) | | ----- | --- | --- | --- | ---- | | Negation| 0.76 | 0.83 | 0.80 | 0.86 | | Synonym | 0.37 | 0.37 | 0.42 | 0.46 | | Second Letter | 0.58 | 0.95 | 0.63 | 1.0 | As shown on three tasks, InstructZero finds better instructions than APE even using only 10\% demos used by APE.