Self-Refine: Iterative Refinement with Self-Feedback

Summary

This paper proposes Self-Refine which uses the same LLM to provide feedback for its output and refine itself to improve output quality. Self-Refine is training-free, using a single off-the-shelf LLM as the generator, refiner and the feedback provider. The authors evaluate Self-Refine across 7 tasks and outputs generated with Self-Refine are preferred by humans and automatic metrics over results of one-step generation. The success of Self-Refine demonstrates that state-of-the-art LLMs can be further improved at test-time.

Strengths

1. The proposed Self-Refine framework is conceptually neat and can be easily combined with different LLMs. Though it still can be viewed as a prompting method, it decouples the refinement of model output into iterative steps instead of focusing on engineering a single prompt. 2. Self-Refine is fairly effective according to the experimental results in Table 1, especially on difficult and uncommon tasks.

Weaknesses

1. Self-Refinement framework relies on prompting for feedback generation and refining outputs. According to examples in Appendix S, the prompt is non-trivial and requires a lot of designing efforts. However, how to design the prompt is not elaborated. Also, Appendix S seems to be unfinished as I see "TODO: Add relevant information for the remaining task" in Line 863. 2. As discussed in Section 2, as an iterative framework, Self-Refine requires a stopping condition. How you design the stopping condition is not clearly stated in the main paper. Also, as there are different designing choices, e.g., using a pre-defined iterative step, using a scalar stop score, comparing refined results and results in the previous iteration, etc., control experiments are necessary. 3. In a high level, I think the contribution of Self-Refine lies in providing a good prompting way to fully tap LLM's potential instead of improving really improving LLM's ability as the authors claim in Line 14-15. For example, Table 1 shows that Self-Refine cannot improve the results in Math Reasoning. I think it may be because the performance of this task is bounded by LLM's reasoning ability and Self-Refine cannot improve such an ability.

Questions

1. I think "Sentiment Transfer" in Line175 refers to "Sentiment Reversal" in Table 2 (I would suggest using the same term for clarity). I'm puzzled why its result in "No feedback" setting is 0. According to Section 3.2, the metric for this task is human preference in A/B evaluation. Does that mean multiple round of generation make the generation result collapse? Do you have any explanation for this? **Updated on 8/11/2023:** The authors' response clarifies my question and some points regarding the weaknesses. Thus, I update my score to 6.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have already discussed the limitations.

Summary

This paper proposes Self-refine, which prompts LLMs to generate feedback on the initial generation and revise the initial response based on the generated feedback iteratively. On 7 tasks, Self-refine leads to performance improvement without any training.

Strengths

- The method is straightforward. - The writing is clear. - The paper provides an in-depth analysis of the proposed method.

Weaknesses

- It is stated that out of 7 tasks, 4 tasks are evaluated by setting GPT-4 as the evaluator for the result of Table 1. Although it is known that GPT-4 possesses order bias, the paper does not handle this bias. Bidirectional evaluation should be conducted to reduce the bias (switching the order between response A and B). - For the result of GPT-4 of Table 1, because the response is refined into GPT-4 preferable response, it is highly likely that GPT-4 would favor the refined response (bias exists). To mitigate this bias, using other LLMs that are not provided by OpenAI such as Claude for either the target model (the model that is being evaluated) or evaluator might be a better solution. - Reliability of the human evaluation setting is questionable. The inter-labeler agreement is missing. Also, author-based human evaluation potentially leads to biased results. - Although the paper states that the authors have experimented with Codex, the result is missing in Appendix F. The main concern of this paper is on the evaluation setting, especially on the results of Table 1. Showing the effectiveness of Self-Refine on additional reasoning tasks based on automatic metrics (such as MMLU, BBH tasks) would enhance the reliability of the evaluation setting.

Questions

- Are there any reason for not conducting a human evaluation for Code Readability (Table 6)? - Any ablation on iterative self-refine? What is the effect of retaining the history of previous feedback? - Any human evaluation on Constrained Generation? It seems that only coverage is measured while other metrics such as coherence might be important. (The generated sentence might be not coherent but the coverage might be high.) - What does stopping indicator refer to in line 86? Any examples of stopping indicator?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have adequately addressed the limitations.

Summary

This paper proposes a Self-Refine framework for improving initial outputs from LLMs. Given an input, Self-Refine generates feedback and refines its outputs iteratively. Without additional training cost or human effort, Self-Refine outperforms baselines with the same large language model that generated output draft previously. Specifically, Self-Refine can be divided into the following three steps: initial generation, feedback, and refinement. These three steps are accomplished through task-specific in-context learning (i.e., few-shot prompting). The authors evaluate Self-Refine on seven diverse tasks (including generation and reasoning tasks). The experimental results demonstrate the superiority of the proposed method, which significantly outperforms baselines on automated metrics or GPT-4-pref (high correlation with human preference on a subset). Subsequent analysis shows that LLMs across different model sizes consistently benefit from Self-Refine. Compared with generic feedback, specific and constructive feedback performs better. Moreover, other analytical experiments have proposed valuable conclusions from different perspectives.

Strengths

1. The impressive experimental results show superior performance to the baseline. 2. The authors have done a wealth of analytical experiments, including the impact of the feedback quality and iterations, effectiveness across different scales of models, etc., which are very meaningful. 3. This well-written paper demonstrates Self-Refine implemented by the authors to generate better outputs with the same LLM.

Weaknesses

1. Implementing iterative few-shot prompting-based Self-Refine may require a high computational cost. However, this is a minor issue since the extent of performance improvement indicates that the cost is worthwhile. 2. Self-Refine doesn't work with weaker models, as it requires the models to have good few-shot modeling or instruction-following abilities.

Questions

The GPT series models used in this paper have good instruction-following abilities. I am interested in how this method performs with zero-shot prompting (only instructions), which would facilitate the community in making trade-offs between performance and cost.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

There is no potential negative societal impact.

Summary

The paper introduces Self-Refine, an approach for improving Large language models (LLMs) through self feedback and refinement. The method does not require further training, and uses a single LLM as the generator, refiner, and feedback provider. The authors evaluate the method on seven NLG tasks, including dialog response generation and mathematical reasoning, using multiple GPT models, i.e., GPT-3.5, ChatGPT, and GPT-4. Experimental results show that Self-Refine greatly improves model performance in terms of both humans and automatic evaluation. **I have read authors' rebuttal as well as other reviewers' comments. I have increased my score from 6 to 7.**

Strengths

1. This paper is well motivated and well written. 2. The proposed method is simple and free of training yet shows great improvement over tasks.

Weaknesses

1. The major concern is that the proposed method can be very costly, in particular when the iteration requires multiple steps. 2. The current paradigm seems to highly rely on the great ability of GPTs, and can not work well for other smaller models, such as Vicuna as discussed in the paper.

Questions

NA

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Thank you for increasing your score!

Thank you for increasing your score! Please let us know if you have any more questions before the end of the discussion period.