Is Self-Repair a Silver Bullet for Code Generation?

Summary

The authors break the problem of program repair into four steps: 1. (LLM) Generate an initial population of programs. 2. Run the programs to obtain error messages. 3. (LLM) Analyze the errors, and generate feedback as to the cause of failure. 4. (LLM) Given the feedback, generate a repaired program. This paper differs from prior work in part because it emphasizes the importance of step (3) -- using feedback to direct the repair. The authors run a number of experiments with different models, explore what happens when a stronger model provides feedback to a weaker one, and also explore what happens when humans provide feedback to a model. The authors also explicitly model the cost of repair. Is running steps (3) and (4) more effective than simply drawing more samples from the initial population? The answer turns out to be "not really". The experiments are well-designed and the authors make several interesting observations. To paraphrase: A. LLMs are currently quite bad at program repair. The best way to get a correct program is to generate a large and diverse initial population, in the hopes that one of the initial programs is "close enough" to work with minimal tweaks. Even a single repair step yields at best marginal improvements. LLMs are thus very different from human programmers; they seem to be unable to iteratively refine and debug programs over multiple steps. B. The main limitation is the quality of feedback. Given feedback from a stronger model, or from a human, LLMs show substantial gains from repair. However, LLMs seem to have difficulty finding problems in the code that they generate.

Strengths

The paper is well-written, and the authors formulation of the problem is insightful. In particular, they do not merely test the accuracy of program repair in isolation, they compare it against the alternative, which is generating new initial programs from scratch. The experiments seem to be well done, particularly the ones which use a stronger model or human to provide feedback to a weaker model.

Weaknesses

The authors do not attempt to fine-tune a model on the task of program repair. Thus their experiments mainly demonstrate that LLMs, *as currently trained*, do not do a good job at the repair task. To be fair, fine-tuning is probably out of scope for this paper, especially since some of they models they test are private, and accessible only via an API.

Questions

Your illustration of the repair-tree is interesting, and it brings to mind the idea of extending this evaluation technique to a proper tree search. You might be able to get better results by using a value network, and focusing only on the nodes of the tree that are most promising for repair, in a manner reminiscent of evolutionary search, with the LLM as mutator. Successful repair attempts, preferably after several iterations of feedback and repair, could also be used to fine-tune the LLM to generate better feedback, and to generate better repair code given the feedback. (See weaknesses, above.) Have you considered any experiments along these lines?

Rating

8: accept, good paper

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Summary

This paper investigates the ability of large language models (specifically, Code Llama, GPT-3.5, and GPT-4) to perform self-repair which refers to the model's capacity to identify and correct mistakes in its own generated code. The paper analyzes the effectiveness of self-repair in addressing code generation errors, specifically on problems sourced from HumanEval or APPS datasets. The findings suggest that the gains achieved through self-repair are often modest and vary significantly across different subsets of data. In some cases, self-repair does not result in noticeable improvements. The paper proposes that this limitation might be due to the model's ability to provide feedback on its own code, and stronger models might enhance self-repair effectiveness. Additionally, the paper explores the impact of providing the model with feedback from human participants, showing that this feedback significantly benefits the self-repair process, even for the advanced GPT-4 model. The paper offers a brief qualitative analysis to shed light on why this human feedback is valuable in improving code repair performance. Overall, the study provides insights into the challenges and potential enhancements of self-repair mechanisms in large language models for code generation tasks.

Strengths

__The paper systematically analyzes the self-repair capability of LLMs.__ While existing works of self-repair mostly argue that self-repair is feasible for LLM, this paper is the first work to systematically study the strengths and weaknesses of self-repair. Specifically, I appreciate the systematic comparison between self-repair and the pass@k performance, though the conclusion might not be generalizable (see Weaknesses), showing that when the computation is comparable, their benefits are also similar, and also suggesting the optimal combination of these two techniques is the most effective. Also, studying the quality of feedback is also a novel perspective to analyze the self-repair capability of LLM __The paper sheds light on the weaknesses of LLM in self-repair, providing clear takeaways and indicating the potential future work in this direction.__ The paper is well written and properly organized, and it sheds light on the weaknesses of LLM's self-repair capability. Such weaknesses can be due to the lack of both data and carefully crafted training strategy for self-repair capability, indicating the future research direction of refining the existing LLM with better self-repair capability.

Weaknesses

I am overall positive regarding this paper, and I appreciate the systematic study the paper performs to quantify LLM's self-repair capability. However, the conclusions are claimed in a strong and general tone, while the study itself is actually limited in scope for two reasons. __The scope of the study is limited to solving isolated programming challenges while ignoring real-world development.__ The study focuses completely on the programming challenges datasets, such as HumanEval and APPs. These datasets have several characteristics that are not realistic in daily development, therefore, though I appreciate the initial conclusions of this paper, these takeaways might not be applicable to a more realistic scenario. First, the samples in the studied datasets are provided with clear and complete problem descriptions, which are not always available in real-world programming practice. One of the main reasons that pass@k works so well in programming challenges is that the expected functionality of the program is fully revealed and clearly explained in the prompt, so the model is able to maximize the diversity within a narrowed semantic space when generating multiple sequences. However, such clearly explained prompts, as natural language, are typically not available during the ongoing development, where the developers start with a very high-level goal and eventually design modules and implement them piece by piece. During this process, the human intent is not always explicitly specified, as docstring or comment, before the code LM is prompted to complete the following code snippets. In these cases, the execution of unit tests provide meaningful feedback to specify and concretize the expected functionalities, which cannot be leveraged by the top-k generation but is valuable guidance for iterative self-repair. Therefore, though I agree with the takeaway that pass@k is comparable to, sometimes better than, self-repair in the programming challenge dataset, such observation might not be realistic for daily development and requires further study. Second the samples in the studied datasets are mostly short and self-contained, missing the complicated data and inter-procedural dependencies. Pass@k explores the breath of each token within the sequence without directional guidance, but such breath or search space exponentially increases with the increase of the code length. The program challenges datasets contain samples mostly up to tens of lines of code, significantly underestimating the complexity of real-world software, which includes hundreds or thousands of lines of code within one single file and maintains complicated dependencies. Generating k sequences blindly without feedback may hardly fulfill the expectation due to the large search space, while execution feedback, such as the indication of a missing third-party library, helps the model quickly locate the problematic code and focus on fixing just that part. Therefore, when the complexity of the program increases, it requires further study to understand whether self-repair is equivalent as top-k generation. __It is not clear whether fine-tuning for self-repair could easily overcome the weaknesses or not.__ This paper focuses on the LLM's self-repair capability only with prompting, without optimizing the model parameters towards the self-repair capability. It is not clear whether a cheap fine-tuning could quickly enable the model's capability of understanding and leveraging the feedback efficiently. Drawing the conclusion that self-repair is not a silver bullet without trying straightforward fine-tuning might be too strong. To conclude, I would encourage the author to consider constraining their conclusions with the study's scope and add a discussion section to mention

Questions

Please explain and address the weaknesses. Otherwise, the paper is well-written and clear.

Rating

8: accept, good paper

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Summary

This paper investigates the sample efficiency of a self-repair approach for LLM based code-generation tasks. It evaluates performance of this approach on HumanEval and APPS dataset using ColdeLLama-13b-instruct, GPT3.5, and GPT4 and provides several insights based on the results: 1) Sampling without repair can perform equally or better than self-repair in almost all sample budgets; 2) Initial sampling diversity is more critical than the diversity of repair samples; 3) Quality of feedback significantly improves the performance of self-repair.

Strengths

* This paper provides several new insights on self-repair for code generation compared to the baseline method of sample generations without self-repair. It shows how sample budget and initial sample diversity could impact the efficiency of code generation. * The investigations on self-repair performance improvement only by improving feedback quality could enable more interesting future ideas. * Overall, the paper is well-written and easy to read. The authors did a great job in highlighting the key limitations of the analysis.

Weaknesses

The experimental results presented in this support the claim around the limitations of self-repair. Interestingly, the findings around overall efficacy compared to baseline somewhat contradicts with the results from Chen et. al. 2023b that shows self-repair could provide significant increase in sample efficiency. Although ‘self-debuggging’ work from Chen et. al. is mentioned in the related work, I think more comparative analysis would strengthen the claim of this paper. Analysis results from more diverse code generation task including datasets other than python language would also be interesting additions to the analysis.

Questions

1. Is there a specific reason to restrict feedback and repair samples to 1 in the analysis of feedback boosting (section 4.2)? 2. Should we expect similar results using the ‘self-debugging’ approach that uses few-shot prompts?

Rating

6: marginally above the acceptance threshold

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

An updated version of the paper addresses feedback, strengthens the results further with additional experiments, and resolves a data processing issue causing incorrect results for Code Llama on HumanEval

We thank all the reviewers for their encouraging comments, thought-provoking questions and detailed feedback! We are especially glad to hear that the reviewers found the paper “well-written” (all reviewers), “properly organized” (PNGf) and that we did a “great job in highlighting the key limitations of the analysis” (Vfe4). When it comes to the methodology, we are pleased to hear that our problem formulation was seen as “insightful” (LPE6) and that the systematic fashion in which we do the comparison was “appreciate[d]” (PNGf). Finally, given the empirical nature of this work we are especially grateful that the reviewers found the experiments “well done” (LPE6), and in particular that our focus on the feedback quality was deemed a “novel perspective” (PNGf) which could “enable more interesting future ideas” (Vfe4). As we enter the second week of the discussion period, we are excited to share an updated version of the paper in which we have addressed the feedback from the reviewers and added several new experiments that strengthen some of our original findings as outlined below. We do want to point out that in the process of reviewing our scripts and running more experiments, we did find one small data processing error that affects one of the Code Llama results; this has now been corrected in the paper. The new results do not affect any of the conclusions of the paper or any of the major claims, but we nonetheless feel it is important for reviewers to be aware of this change. Concretely, the changes made to the paper are: - In response to feedback about the language sometimes being a bit too strong/general, as well as the discussion we’ve had with PNGf01 about how real-world software engineering differs from the tasks we’ve focused on here, we have replaced the second paragraph in the Limitations section with a discussion of how this work relates to more realistic Software Engineering tasks. We have also made minor revisions to the introduction to make it even more explicit to the reader early on that we are studying only self-contained Python programs in this work, as well as some minor wording changes throughout to clarify the scope of our claims. These changes are in **blue**. - We have added the following extra experiments: - Code Llama baseline and self-repair on APPS - Code Llama + GPT-3.5 on APPS - Code Llama + GPT-4 on HumanEval - The extra experiments we have added agree with and further strengthen our key findings: - Code Llama also does not appear to benefit from self-repair on APPS (much like GPT-3.5) - Code Llama does see performance improvements when given feedback from GPT-3.5 - On HumanEval, boosting Code Llama with feedback from GPT-4 significantly improves performance compared to when doing so with GPT-3.5, further suggesting that the feedback alone can make a tremendous difference. - The data processing error mentioned above was harming Code Llama’s reported self-repair performance on HumanEval. This error has now been fixed; we find that Code Llama does benefit from self-repair on this dataset, but that the feedback stage still remains a bottleneck, as illustrated by the significant improvements of Code Llama when paired with GPT-4. - We have updated the Experiments section and appendices to include these results; all changes are in **green**, including updated figures. - To make room for these additions, we have done some very minor word-smithing throughout the text. We once again thank the reviewers for their extremely helpful comments and suggestions. We believe that these changes have further strengthened what the reviewers already appear to have found to be a strong paper, and hope that the reviewers also found the individual responses insightful. **If the reviewers have found this discussion satisfactory, we now kindly ask them to consider updating their scores accordingly. If, on the other hand, they believe further discussion is needed, we eagerly await their thoughts and comments!** 🙂