Guiding Large Language Models via Directional Stimulus Prompting

Summary

This paper proposes a Directional Stimulus Prompting method to provide fine-grained guidance for the output of large language models (LLMs). This method introduces a small trainable policy model (e.g. FLAN-T5) to generate hints for each query to guide LLMs towards desired outputs. This policy model can be trained via a standard paradigm including supervised fine-tuning on pseudo-labeled data and reinforcement learning (RL) to optimize the expected reward. Experiments on summarization and dialogue generation tasks show the effectiveness of the proposed method.

Strengths

1. This paper proposes a direct and feasible solution to guide black-box LLMs towards better generation results, which is widely applicable with the rapid development of LLMs. 2. This paper is well-written and easy to follow.

Weaknesses

1. The technical novelty of the proposed method is limited. The training of the policy model in this paper is a standard paradigm including supervised fine-tuning and reinforcement learning. The techniques used in RL including dynamic adjustment of the coefficient and NLPO are all borrowed from existing works. Thus, I feel that the main difference falls into the usage of LLMs as an evaluation function in the reward model, which is devised for adapting this RL algorithm to the scenario of guiding LLM. Thus, the novelty of this design is mainly on the applicational side. 2. During supervised fine-tuning, the authors heuristically select the “pseudo-stimulus” for each input. The pseudo-stimulus indicates keywords / dialogue acts for summarization / task-oriented dialogue generation, respectively. I wonder whether there is a general principle to select pseudo-stimulus especially in open-ended text generation tasks, since recently proposed LLMs such as ChatGPT are mainly applied to these tasks. 3. The benchmark datasets in this paper only contain CNN/DM for summarization and MultiWOZ for task-oriented dialogue generation, which are not convincing enough to evaluate the performance of the proposed method based on LLMs. The authors should conduct experiments on broader tasks and datasets. Also, I’m curious about the motivation to choose MultiWOZ. In Section 3.2, the authors say that LLM-based chatbots such as ChatGPT face challenges in handling task-oriented dialogue generation because this task requires the chatbot to respond based on reliable information from API calls or database queries. But the proposed method still cannot interact with APIs or databases. The experimental setting of MultiWOZ seems like end-to-end response generation (i.e. generating the system response given the dialogue history), which is similar to open-domain conversations. 4. From [1], automatic evaluation metrics such as ROUGE cannot reliably evaluate the quality of LLM-generated summaries. Thus, human evaluation should be added to strengthen the experimental results. [1] News Summarization and Evaluation in the Era of GPT-3.

Questions

I have included my questions in the weaknesses part.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

The authors should discuss more on the risk of guiding LLMs to generate unethical contents and how to avoid it via the policy model.

Summary

This paper introduces a novel approach to effectively guide black-box LLMs towards generating desired outputs. The proposed approach involves utilizing a relatively small policy model to generate "directional stimulus," which serves as specific information to assist LLMs in performing tasks such as summarization and task-oriented dialogue. The policy model is trained using rewards obtained from evaluating LLM outputs. Notably, this approach requires only a small amount of training data and eliminates the need to train the entire model.

Strengths

- The writing is clear, making it easy to grasp the motivation and contribution of the paper. - The proposed approach is simple, which suggests its potential for widespread adoption across various domains. - The experimental results effectively demonstrate the effectiveness of the approach. Notably, in Figure 3, the model leveraging DSP achieves superior performance in terms of BLEU, METEOR, and BERTScore, despite Rouge being used as the reward metric. - The fact that the authors have released their code is highly valuable, as it facilitates future research and allows for the replication of their findings.

Weaknesses

- I have a concern regarding the generalizability of the method, as the experiments primarily rely on a single prompt per task. - It is widely acknowledged that LLMs heavily rely on the structure and format of the prompt, which raises the question of whether this technique would be effective with different prompt formats. - This approach would hold even greater value if it could consistently yield improvements regardless of the choice of prompt format, demonstrating its robustness. - Additional analysis about generated clues will be helpful to get lessons from this work. For example, - How does the result change if the number of clues is increased/reduced? - What kind of clues are generated considering the semantics/contents of the input (visualization will be helpful) - The method is not intuitive to me, because all the information that is required to perform the task is already contained in the input. Does the result mean that LLMs are not sufficient to extract information from naive and complex inputs? It will be helpful if the authors provide thoughts about this phenomenon.

Questions

- It would be valuable to know the exact API cost associated with training the reward model. The authors seem to have made efforts to minimize API calls, and providing this information would offer additional details for other researchers to reproduce the study accurately. - In relation to the aforementioned weakness, it would be interesting to investigate how the model's performance is affected by changing the prompt template. Understanding the impact of prompt variations would provide insights into the robustness of the approach. - Regarding Figure 1, it appears that the part describing what is "highlighted in blue" is missing. Including this information would clarify the intended meaning. - As a suggestion, exploring the possibility of allowing the model to "edit" the prompt could potentially offer greater flexibility and a wider search space, leading to improved performance. This suggestion aims to enhance the approach by incorporating additional avenues for improvement.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors' discussion of the limitations in the paper is not sufficient. Addressing the suggested weaknesses and questions in reviews would significantly enhance the insights provided by this paper.

Summary

In this paper, a prompting framework called Directional Stimulus Prompting (DSP) is proposed which provides a more fine-grained guidance and control over LLMs by adding directional stimulus into the prompt. These directional stimulus or hints are generated by a small tunable model which is fine-tuned using supervised learning and reinforcement learning. The performance of the model is tested on summarization and dialogue response generation tasks.

Strengths

S1: It is interesting that they fine-tuned and used a small language model to improve the performance of larger language models such as ChatGPT. This approach cleverly bypasses the constraint of being unable to fine-tune large language models. S2: In both experiments, they demonstrated the effectiveness of their approach by showing the improvement achieved through their framework. S3: The paper is well written and has a good flow. S4: A good number of related works are covered in the Related work section.

Weaknesses

In their experiments, they only show the results for the flan-T5-large model as a model for generating directional stimulus. They could have also included the performance of other fine-tuned models in their experiments.

Questions

See weakness section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

One of the good points that they motioned is that their framework could be used to guide LLMs to generate harmful or biased contents.

Summary

This paper introduces a interesting module named 'Directional Stimulus Prompting' (DSP). This innovative module functions by generating cues or hints to aid black-box Large Language Models (LLMs) in response generation. For instance, in a summarization task, providing keywords can guide the LLM towards generating more accurate and relevant answers. The uniqueness of this module lies in its ability to be fine-tuned through both supervised learning and reinforcement learning techniques. In reinforcement learning, automatic metrics (reward) are applied to responses generated from black-box LLMs, which have been influenced by the DSP. The results from experiments confirm that both the supervised fine-tuning and reinforcement learning approaches to DSP can significantly enhance the performance of black-box LLMs. This finding underscores the potential of DSP as a powerful tool for improving LLM response generation.

Strengths

This paper introduces a compact yet powerful module known as 'Directional Stimulus Prompting' (DSP). Its function is to generate hints that enhance queries posed to black-box Large Language Models (LLMs). Two distinct approaches for fine-tuning this DSP module are presented by the authors: supervised fine-tuning and reinforcement learning. The effectiveness of both supervised fine-tuning DSP (SFT DSP) and reinforcement learning DSP (RL DSP) is confirmed through experimental results, underscoring the utility of this novel module in enhancing LLM query performance.

Weaknesses

Currently, automatic metrics are employed for evaluation, as well as for the reinforcement learning (RL) tuning of the Directional Stimulus Prompting (DSP) module. However, presenting these metrics exclusively in the final analysis could restrict the comprehensive assessment of performance. It may be more insightful to incorporate other evaluation metrics into the analysis. This could include human evaluation, entity word extraction, and GPT-based evaluations. Such a diversified metrics approach could potentially provide a more holistic view of the performance improvements achieved through the use of the DSP module.

Questions

Could the "few-shot prompting" or "in-context learning" potentially enhance the effectiveness of the Directional Stimulus Prompting (DSP) module?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This paper introduces Directional Stimulus Prompting (DSP), a new prompting framework that introduces directional stimulus into the prompt, which could provide black-box LLMs with fine-grained and query-specific guidance toward the desired outputs. The experiments on summarization and dialogue response generation tasks demonstrate the effectiveness of this approach. Notably, on the MultWOZ dataset, our framework enables ChatGPT to achieve a remarkable 41.4% improvement in its combined score with only 80 dialogues.

Strengths

1.It is quite novel that the paper proposed DSP, a prompting framework for guiding black-box LLMs toward desired outputs, which combined Supervised fine-tuning and Reinforcement learning to further optimize model. 2.The experiment setting is quite detailed, since this paper used varying numbers of training samples from the datasets and evaluation metrics for ease of display and comparison.

Weaknesses

1.There is too little dataset in the experiment section. Both tasks were conducted on a single dataset and cannot fully demonstrate the effectiveness of the framework. 2.On Page 5 Line 151, the experiment section of the paper is not enough to prove this conclusion. It is not rigorous to say that the framework can be flexibly applied to various types of LMs and generation tasks, just by conducting experiments on two tasks.

Questions

None

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

None

Response to Rebuttal

Thanks for your rebuttal. My main concern is still about the technical novelty. As I mention in my original review, the novelty of the method is mainly on the applicational side, which means that applying a general method (i.e., SFT+RL) to deal with a new scenario (i.e., guiding black-box LLM outputs towards desired outcomes for specific tasks) is the main focus. I think the applicational papers can also have novel contributions if the authors propose techniques on how to apply the method to the specific task considering the task's characteristics. However, the techniques in this paper are nearly borrowed from existing works, which provide few insights on solving this task.

Response to Authors' Rebuttal

Thank you to the authors for their rebuttal. The explanations provided have addressed some of the weaknesses I highlighted. As a result, I have revised my score to 7 (Accept).