In the domain of code generation, self-debugging is crucial. It allows LLMs to refine their generated code based on execution feedback. This is particularly important because generating correct solutions in one attempt proves challenging for complex tasks. Prior works on self-debugging mostly focus on prompting methods by providing LLMs with few-shot examples, which work poorly on small open-sourced LLMs. In this work, we propose LeDex, a training framework that significantly improves the self-debugging capability of LLMs. Intuitively, we observe that a chain of explanations on the wrong code followed by code refinement helps LLMs better analyze the wrong code and do refinement. We thus propose an automated pipeline to collect a high-quality dataset for code explanation and refinement by generating a number of explanations and refinement trajectories from the LLM itself or a larger teacher model and filtering via execution verification. We perform supervised fine-tuning (SFT) and further reinforcement learning (RL) on both success and failure trajectories with a novel reward design considering code explanation and refinement quality. SFT improves the pass@1 by up to 15.92% and pass@10 by 9.30% over four benchmarks. RL training brings additional up to 3.54% improvement on pass@1 and 2.55% improvement on pass@10. The trained LLMs show iterative refinement ability and can keep refining code continuously. Lastly, our human evaluation shows that the LLMs trained with our framework generate more useful code explanations and help developers better understand bugs in source code.
Paper
Similar papers
Peer review
Summary
The paper addresses the goal of self-debugging of generated code, while also explaining it. The approach is to: (a) sample code outputs to natural language inputs, and keep only the wrong code outputs according to unit tests; (b) sample **refinements** ("fixes") to the wrong code outputs and test those refinements using unit tests, to get both correct and incorrect refinements; (c) the authors train an LLM using SFT and RL on those correct and incorrect refinements. The resulting trained LLM is shown to provide empirical gains, especially when allowing it to self-refine its outputs at test time, with the test-time unit tests.
Strengths
* The proposed pipeline provide strong empirical gains * The authors experiment with 3 base LLMs (StarCoder-15B, CodeLlama-7B, CodeLlama 13B) and 2 "teacher" LLMs (gpt-3.5-turbo, CodeLlama-34B) * The authors experiment with multiple datasets: they train on MBPP, APPS, CodeContest, and test on HumanEval and MBPP.
Weaknesses
* The paper is mostly applicative, and includes a mix of known techniques. I feel that the paper is low on conceptual novelty. The concept of self-debugging and verifying correctness using unit tests was introduced by [Chen et al., 2023](https://arxiv.org/pdf/2304.05128) (although only using prompting), and RL using the signal coming from the unit tests was done in several papers (that the authors cite). * It feels like there are so many different techniques and heuristics involved, that it is hard to pinpoint the exact contribution of each of them. For example: the design of the explanation score as $CosSim(RoBERTa(e), RoBERTa(ec))$, the exact hyperparameters of the reward design (e.g., $\frac{50 · S_{ex}(e) 35}{3}$), and the exact design of $S_{cb}$ using CodeBLEU. The use of explanations before fixing the code is a form of Chain-of-Thought (Wei et al., 2022), or the "feedback" in Self-Refine (Madaan et al., 2023). * Further, teaching the models to self-debug is done using **larger models**. That is, the refinements are sampled from larger models such as gpt-3.5-turbo, CodeLlama-34B, and then these refinements are used to train the smaller StarCoder-15B, CodeLlama-7B, CodeLlama 13B models. This adds a dimension of distillation (from large models to small models), and further makes it difficult to pinpoint the exact source of contribution. * The approach is not compared to any related work, or to the teacher models themselves. * Position in literature - Although many papers are cited, I feel that the paper does not position itself well in the literature. I do not remember exactly what did each related work do, but the paper does not help me understand the differences and its novelty compared to the related work. For example, what's the difference between this paper and [18] and [20]? They seem very similar, but the Related Work does not highlight the novelty over them. * Another new paper that was not cited: [Ansong et al., NExT: Teaching Large Language Models to Reason about Code Execution, ICML'2024](https://arxiv.org/pdf/2404.14662). I am not interested in the authors just citing the paper as an additional number between brackets, I am interested in discussing the actual differences and novelty over that paper.
Questions
1. At test time, when evaluating the SFT/RL models: does the model see the execution results of its generated test code before refinement? That is, can the model use the unit tests of the test examples as well, or are unit tests used only at training time? 2. In the definition of $S_{cb}$ - if we rely on unit tests to verify correctness, why do we need to encourage the refinement to be similar (in terms of CodeBLEU) to **all** the possible correct refinements? 3. RL training on benchmark data may over-specialize on their specific domain, while degrading the general coding abilities of the LLM. After all, these benchmarks are only benchmarks, and over-specializing on them may hurt the usability of the model in practical use. For example, have the authors checked whether applying their SFT+RL hurts the perplexity on general code? To summarize, I think that the paper presents strong empirical gains, but the scientific novelty is low, as the contribution is mostly applied. I thus vote for a borderline reject.
Rating
7
Confidence
4
Soundness
3
Presentation
3
Contribution
2
Limitations
N/A
Summary
This work proposes a novel framework to enhance the self-debugging capabilities of smaller language models that do not benefit much from self-refine or other prompt-based debugging approaches. Sampling incorrect code samples produced by LMs, they pass the execution feedback on these to GPT-3.5/4 and prompt it to explain the reason for the errors and propose code refinements. Accurate refinements are used to fine-tune and create a code-correcting model. This is further enhanced with a PPO based learning from a novel reward assignment mechanism accounting for both explainability and code refinement. Overall, they demonstrate the importance of having explanations for incorrect codes and how RL can be used to enhance the debugging ability of models to show superior performance across benchmark datasets.
Strengths
- Technically solid - This work addresses a significant issue of little coding improvement from self-refinement prevalent in smaller LMs - The reward setup incorporating both code refinement and explainability in PPO is novel and shows good gains
Weaknesses
Nothing major, future evaluation on datasets like APPS etc. could be beneficial to understand the impact on harder tasks
Questions
- Was there any reason behind choosing the range of score to be [-5,5]? - It seems in the bigger models tested (\geq 13B) the refine itself is quite effective, any intuitions on this scale effect?
Rating
7
Confidence
5
Soundness
4
Presentation
3
Contribution
3
Limitations
Yes
Summary
The authors teach language models to better self-debug and explain code. Particularly, they utilize code explanations in the repair process where explanations are generated before refining the programs. This is accomplished via training the models with SFT and RL on data curated from different sources and model generations with test case based rejection sampling.
Strengths
The paper is nicely written with enough details about experiments. Improving code repair capabilities of LLMs is an important problem and the authors propose a rejection-sampling-based technique to automatically curate repair trajectories from LLMs. The associated ablations are useful and convey useful insights.
Weaknesses
Weak results * Benefit of explain + refine over just refine. The proposed explain + refine approach, associated loss on explanations does not seem to be improving performance. Infact, Expl.+Refine rows sometimes do worse than the only Refine rows. These results highlight a lack of benefit from adopting this approach. While I understand that having associated explanations is appealing and perhaps might even provide models with more inference-time-compute before performing refinements, I think the current approach or experiments does not convey that. The associated human evaluations (Table 7) also point to similar findings. * Optimizing single-turn vs multi-turn performance. Finally, a lot of the instruction-tuned variants of the models used in this work claim better single-turn HumanEval and MBPP performance. I wonder how the findings would change if they started from a strong instruction-tuned model and improved its repair performance. For instance, the OpenCodeInterpreter-CL-7B model shows performance improvements of 3 points from execution feedback on HumanEval (from 72 to 75) due to strong instruction tuning data pushing pass@1 to 72. Perhaps to a larger point, the effect of repair depends on the choice of the underlying model and broader data mixture which this paper does not study. * Choice of evaluation datasets. Since the authors use competition programs (like APPS or CodeContests) in their study, perhaps it is fair to also evaluate the models on competition programming datasets. * Missing related works. [1] is also pertinent to LLM reasoning. [2] and [3] also train and release open LLMs on repair trajectories. [3] in fact uses a similar explanation format. [1] Reflexion: Language Agents with Verbal Reinforcement Learning [2] OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement [3] Advancing LLM Reasoning Generalists with Preference Trees
Questions
* Table 1 further details. I anticipate that many problems in the training sets remain unsolved post-refinement, particularly for APPS and CodeContests which are more challenging benchmarks. Can the authors list details about problems beyond number of solutions.
Rating
5
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
Limitations are discussed adequately (besides some of the above-mentioned weaknesses)
Thanks for the response. > The reviewer might have some misunderstanding of the explanation evaluation ... SFT/RL training significantly improves the explanation quality From what I can see, the absolute ratings from developers is below 3 on average on a 1-5 scale. While the performance improvement from untrained model baseline is considerable, the absolute scores are still not great. I suspect a reason why absolute numbers are low is that not all explanations lead to refinement and the average rating might be higher in the second scenario. > The results in Appendix A.4.1 show that even when training with strong code instruction data, the model does not obtain self-debug capability out-of-the-box and it only refines up to 3% of its wrong solutions Perhaps I was not clear enough -- this work depicts considerable mutliturn improvements from RL training (48% to 57% pass@1 on HumanEval for CodeLLama-7B). My concern is that if the authors start their RL training with a stronger model (say OpenCodeInterpreter-CL-7B ) which already has 70+ pass@1 on HumanEval will the RL training be as effective? For example, OpenCodeInterpreter-CL which trains models with multi-turn SFT dataset only achieves 3% improvements from 72 to 75. This makes it challenging to interpret the performance improvements achieved in this paper.
Thank you for following up. ## Human Rating Below is the breakdown of the number of explanations having the score in each range. Although the overall average score is below 3, SFT and RL can generate 24 and 27 explanations with scores higher than 3 (50 samples in total). | Score | Prompt | SFT | RL | GPT-3.5-Turbo | |-|-|-|-|-| | 4.5 <= score | 1 | 3 | 5 | 13 | | 4 <= score | 1 | 10 | 11 | 21 | | 3.5 <= score | 3 | 16 | 17 | 26 | | 3 <= score | 7 | 24 | 27 | 35| If we look at poor explanations, the RL model generates a similar number of explanations with “score <= 1.5” compared to GPT-3.5. | Score | Prompt | SFT | RL | GPT-3.5-Turbo | |-|-|-|-|-| | score == 1 | 19 | 6 | 4 | 6 | | score <= 1.5 | 33 | 14 | 10 | 9 | | score <= 2 | 39 | 21 | 21 | 14 | | score <= 2.5 | 43 | 26 | 23 | 15 | We also find that human annotators tend to be more harsh and give lower scores than GPT-4. Figures 12 and 13 include examples of explanations with “score 4”, which we think the model explains pretty well. ## SFT/RL Improvement There seems to be some misalignment from the number you mentioned. If we look at Table 2, HumanEval Pass@1 using CodeLlama-7B. With code explanation and refinement (Expl. + Refine.), prompting’s Pass@1 is 40.13%, SFT’s Pass@1 is 52.98%, RL’s Pass@1 is 55.84%. So SFT increases it by about 13%, and RL (on top of SFT) further increases it by about 3%. And also by looking at Table 9, with only single-turn data (MBPP, APPS, CodeContests, MagiCoder’s data), the Pass@1 on HumanEval is 43.88%. SFT (with multi-turn refinement data) can still get 9% improvement. For OpenCodeInterpreter, it has been trained on high-quality data collected using GPT-4, so the base Pass@1 is already high enough (72%), which may nearly reach the model’s limit and leave less space for mult-turn SFT refinement to improve. That could be why multi-turn SFT only improves 3%. **We think this is not contradictory with our results.** Our focus is on how to train LLM to self-debug starting from standard code generation training data (MBPP, APPS, CodeContests). Thus we only use GPT-3.5 as the teacher to collect multi-turn data. We also test by using CodeLlama-34B as the teacher (Table 5), and even using CodeLlama-7B itself to self-bootstrap multi-turn data (mentioned in global response). Our method can work without relying on GPTs. This work is not competing on data quality, so the final results are not supposed to outperform OpenCodeInterpreter. But we would like to add OpenCodeInterpreter as related work and discuss it in our paper.
Thank you for the quick response. > If we look at Table 2, HumanEval Pass@1 using CodeLlama-7B. With code explanation and refinement (Expl. + Refine.), prompting’s Pass@1 is 40.13%, SFT’s Pass@1 is 52.98%, RL’s Pass@1 is 55.84%. So SFT increases it by about 13%, and RL (on top of SFT) further increases it by about 3%. > > And also by looking at Table 9, with only single-turn data (MBPP, APPS, CodeContests, MagiCoder’s data), the Pass@1 on HumanEval is 43.88%. SFT (with multi-turn refinement data) can still get 9% improvement. Apologies for the misquoted numbers and thanks for the clarification. I will update my rating.
Summary
This paper proposed a pipeline to obtain code explanation and refinement data from a stronger model (mainly GPT-3.5 with CodeLLaMA as ablation) to train weaker models (i.e., StarCoder-15B, CodeLLaMA-7B/13B) using SFT and RL methods. More specifically, it uses the weaker models to sample the incorrect solutions and use the stronger models to give code refinement and explanations. It also designed reinforcement learning rewards specifically for the code refinement and explanation tasks. Experiments are conducted on MBPP and HumanEval, as well as their EvalPlus versions, and results show that the both SFT and RL yields improvements over the baselines, while the improvements from RL are relatively marginal compared to those of SFT.
Strengths
S1. The tasks of code refinement and explanation are increasingly important due to the popularity of using language model agents for coding tasks. And this work shows a way to reliably improve the performance of LLMs in these two tasks, which may be useful for broader domains such as code editing and reasoning; S2. The RL reward design and ablations could be useful for further research on using RL for code explanations and refinements; S3. The paper is well-written, with clear motivations, details of methodology and comprehensive experiments.
Weaknesses
W1. Something slightly disappointing is that this work chooses to generate training data **from a stronger model**, which classifies it into the category of distillation, while the entire pipeline could have been done with the same LLM to explore the potential of self-improvement; W2. While the paper focus a lot on the design of the RL method (which seems to be a big part of the contribution from my understanding), the actual improvements yield by the RL methods are quite marginal. However, I do note that at least the performance does not decrease for the most cases; W3. There are some baselines experiments and ablations that can be added to make the evaluation part stronger (see questions below).
Questions
Q1. According to Olausson et al., 2023 on self-repair, after $k$ rounds of self-refinement after the initial code generation, the success rate is often less than simply pass@$k+1$. And also note that the sampling can be done in parallel while the iterative refinement can only be done sequentially. Have you compared the refinement results with pass@$k+1$ to see if self-refinement yields any benefit before / after the SFT/RL training? Q2. From Fig. 3(a), it seems to me that the CodeBLEU score is not a good metric as it can barely separate the correct and wrong outputs distribution-wise, is there any reason for CodeBLEU to still be factorized into the reward function despite this? Q3. Can you comment on the reliability of using the RoBERTa embedding for measuring the similarity of the explanations? Are there better ways to do this? Q4. From Tab. 2, it seems that after SFT, the "Init." performance also significantly improved, does this means that **only** training on code refinement and explanation can also improve code generation? Q5. (Pertain to W3) Have you tried to use the same model as the LLM to create the training data? Q6. The training data are created on top of APPS and CodeContest as well, but why are those two datasets not used in evaluation? **References** Olausson, Theo X., et al. "Is Self-Repair a Silver Bullet for Code Generation?." The Twelfth International Conference on Learning Representations. 2023.
Rating
8
Confidence
4
Soundness
3
Presentation
4
Contribution
3
Limitations
N/A
Summary
Authors propose a framework to perform SFT and RL training to achieve superior performance in generating code with open code LMs on the self-debugging task. They leverage the test suite present in benchmarks like APPS, CodeContests to obtain execution feedback for model refinements on CodeLM generated code. The resulting dataset is then used for SFT training, followed by RL training that can also leverage failed trajectories besides the successful ones (where the model succeeded in generating a code that passes all unit tests). Authors show promising gains for open weight models in the model's capability to self-debug after training with their proposed methodology.
Strengths
- Authors present a strong motivation for this work (Lines 43-52) on achieving strong code generation performance with open weight models. - This work makes a strong contribution in the form of a framework to construct data for SFT and RL training of a model that can perform self-debugging after explaining faulty code. Authors propose a clever way of leveraging execution feedback in constructing their datasets - The reward construction based on environment feedback is a particularly important contribution that significantly adds to the novelty of this work. To my knowledge prior work hasn't utilised environment feedback in this manner. - Convincing results that confirm the utility of training open codeLMs on the task of self-debugging.
Weaknesses
- While the experiments and analysis of the results and datasets are fairly exhaustive, I believe the choice of RL algorithms should be justified by considering or eliminating alternatives like preference optimisation using DPO or KTO. I'd suggest at least adding a discussion on the pros and cons of preference learning compared to the RL setup that the authors advocate in this work. - Some missing baselines: teacher model (GPT-4/3.5/CodeLlama35B) performance is missing in Tables 2 and 5. Authors do not discuss the persisting gap in performance in any between the models used in creating the datasets and the performance their approach attains on the benchmarks used. - A very relevant related work (Teaching LLMs to Self-Debug https://arxiv.org/abs/2304.05128) mentions gains in sample efficiency as one of the major benefits of performing self-debugging/refinement. I could not find a discussion or results on this aspect for the fine-tuned models presented in this paper.
Questions
- APPS and CodeContests are used to train the CodeLMs, but I could not find evaluation on the APPS-test or CodeContests test set. Could you explain this choice? - Can you provide details on the number of GPU hours required in the experiments? - Have the authors considered training a separate model using the SFT and RL techniques to solely solve code refinement for code generated from the base model? - What do the authors think about the generality of this method to improve on attributes beyond correctness in refining code? e.g. readability, performance and secureness of generated code. - I'm curious how the proposed approach would compare against a simple baseline where the model is SFT-trained on the final refinement collected in your training set. The current setup involves using problem description $x$ and ground truth code $y$, test suite $T$ to generate a synthetic code solution $y'_l$ that fails and is explained by $x_l$ followed by a successful refinement $y'_w$. Your approach then trains the model to generate $x_l, y'_w$ given $x, y, y'_l$. A simple baseline to compare against could involve training the model to generate $y'_w$ given $x$. This would confirm the value in framing code generation as a explanation + refinement task.
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
Briefly discussed by the authors in Section 5. Could be expanded to include a discussion on other aspects of code refinement not covered in the paper, and acknowledging gaps if any in performance of open models trained with this method and closed models.
Response to authors
Thank you for your response. Can you please edit your response and mention which part of my review does each part in your response refer to? For example, I did not ask for a *comparison* to the teacher models. I also asked: >>2. In the definition of $S_{cb}$ - if we rely on unit tests to verify correctness, why do we need to encourage the refinement to be similar (in terms of CodeBLEU) to **all** the possible correct refinements? The authors' response seems to be answering a different question. My emphasis was on the word "all" - why does it make sense to encourage the refinement to be similar to **all** the possible correct refinements? What if there are multiple ways to solve the same problem? Regarding generalization, the authors mention that the perplexity increases from 1.457 to 1.597, which might be evidence that the model's coding ability is indeed hurt. How can we be convinced that the model does not over-specialize on specific domains?
Thank you for following up. We re-organized the rebuttal so that the response to each question is clear. Due to character limits, we abbreviate the reviewer’s questions, and refer some questions to our first response. Below, we focus primarily on addressing the reviewer’s new questions. ## 1. Paper novelty Reviewer: “The paper is mostly applicative, and includes a mix of known techniques. I feel that the paper is low on conceptual novelty.” Response: Please refer to the “Paper novelty” in our first response. ## 2. Position in literature Reviewer: “Position in literature - Although many papers are cited, I feel that the paper does not position itself well in the literature.” Response: Please refer to the “Position in literature” in our first response. ## 3. Self-taught refinement Reviewer: “This adds a dimension of distillation (from large models to small models), and further makes it difficult to pinpoint the exact source of contribution.” Response: Please refer to “Self-taught refinement” in our first response. ## 4. Comparison to the teacher models or related work Reviewer: “The approach is not compared to any related work, or to the teacher models themselves.” Response: The prompting baseline mentioned in our paper refers to the related work [1], and we compare with prompting method in our experiments. We provide the comparison with the teacher models CodeLlama-34B and GPT-3.5-Turbo on the self-debugging setup in the global rebuttal pdf. Comparing it with Table 2 and Table 10 in the paper, we see that with CodeLlama-34B as the teacher, CodeLlama 7B SFT/RL achieves close to CodeLlama-34B self-debugging performance, while CodeLlama-13B SFT/RL significantly outperforms the CodeLlama-34B teacher (e.g. in HumanEval+ pass@1 56.24% vs 48.51%). ## 5. Unit Test at test time Reviewer: “At test time, when evaluating the SFT/RL models: does the model see the execution results of its generated test code before refinement?” Response: please refer to “Unit Test at test time” in our first response. ## 6. Reward Design Reviewer: “why does it make sense to encourage the refinement to be similar to all the possible correct refinements?” Response: If we only consider one correct solution, there will be correct refinement that uses a different way but get very low CodeBLEU score. And it is also unreasonable to train the model to follow only one correct solution. Our solution does not penalize the model too much when it uses a different way to solve the problem, as long as there exist correct solutions using the similar way. Besides, the CodeBLEU score is mainly used to densify the reward distribution, unit test is more important to separate the wrong and correct solutions. ## 7. Generalization Reviewer: “RL training on benchmark data may over-specialize on their specific domain, while degrading the general coding abilities of the LLM.” Response: We have already considered the practical usage of the proposed training, and all of our experiments in the paper use rather comprehensive data, not only the self-debugging data we collected but also the original code generation data provided in the MBPP, APPS, CodeContests training set and Magicoder dataset [2] to avoid over-specialization. Besides, to avoid overfitting, we use a large batch size so the model weights are only updated for about 2000 steps.. We test our models’ perplexity on general code, e.g., 10000 samples from BigQuery Python code. The CodeLlama-7B pre-trained model’s perplexity is 1.457, the SFT model’s perplexity is 1.597, and the RL model’s perplexity is 1.599. Both are just slightly higher than the pre-trained model. **Code generation is one of the most important code tasks for LLMs and our trained model is much better than the pre-trained model on it.** **The SFT/RL models’ perplexity on the pretraining data being higher than the pretrained model doesn’t mean they generalize worse. The SFT (or instruction-tuned) LLMs typically have higher perplexity than the pre-trained foundation model, since they learn to follow human instructions.** We also test the instruction-tuned CodeLlama-7B released by Meta (https://huggingface.co/codellama/CodeLlama-7b-Instruct-hf), which gets a perplexity of 1.682, even higher than ours. We generally don’t see concerns regarding the higher perplexity caused by instruction-tuning, because instruction-tuned models follow users’ instructions better and are more useful in developing AI-assistant. Reference: [1] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug, ICLR 2024. [2] Wei, Y., Wang, Z., Liu, J., Ding, Y., & Zhang, L. (2024). Magicoder: Empowering code generation with oss-instruct. In Forty-first International Conference on Machine Learning.
Thanks for the response
I'd like to thank the authors for the detailed response, especially the additional experiment results. I find the additional results interesting and promising, and I think the argument on using CodeBLEU stabilizes training makes a lot of sense. I think adding the results / discussions from at least 1+2+3 from above would make the paper more interesting and stronger, so I hope the authors would add them in the next version of the paper. I have improved my score accordingly, good luck!
Decision
Accept (poster)