Large language models (LLMs) have shown remarkable progress in code generation, but their generated code often suffers from inefficiency, resulting in longer execution times and higher memory consumption. To address this issue, we propose \textbf{EffiLearner}, a self-optimization framework that utilizes execution overhead profiles to improve the efficiency of LLM-generated code. EffiLearner first generates code using an LLM, then executes it locally to capture execution time and memory usage profiles. These profiles are fed back to the LLM, which then revises the code to reduce overhead. To evaluate the effectiveness of EffiLearner, we conduct extensive experiments on the EffiBench, HumanEval, and MBPP with 16 open-source and 6 closed-source models. Our evaluation results demonstrate that through iterative self-optimization, EffiLearner significantly enhances the efficiency of LLM-generated code. For example, the execution time (ET) of StarCoder2-15B for the EffiBench decreases from 0.93 (s) to 0.12 (s) which reduces 87.1% the execution time requirement compared with the initial code. The total memory usage (TMU) of StarCoder2-15B also decreases from 22.02 (Mb*s) to 2.03 (Mb*s), which decreases 90.8% of total memory consumption during the execution process. The source code of EffiLearner was released in https://github.com/huangd1999/EffiLearner
Paper
Similar papers
Peer review
Summary
The paper aims to address the inefficiency issue of code generated by LLMs in terms of execution time and memory consumption. The authors propose a framework called Self-Optimization based on OverheAd Profile (SOAP), which improves the efficiency of LLM-generated code by iteratively refining it based on execution overhead profiles. These profiles capture the code's execution time and memory usage, which are then fed back into the LLM to optimize the code. The paper demonstrates the effectiveness of SOAP through extensive experiments on various benchmarks and selected LLMs. Results show reductions in execution time and memory usage, highlighting SOAP's ability to enhance code efficiency without compromising correctness.
Strengths
* The paper is well presented and the targeted problem of code inefficiency is well motivated. * The evaluation covers a good number of Python coding tasks and LLMs. * The authors mentioned that the code of the work will be open-sourced.
Weaknesses
* The baseline selection can be largely improved. The current baseline in the paper is to ask LLMs to directly generate code -- it is not the models' fault to generate non-optimal code when they are not asked to. At least two reasonable and simple baselines should have been included: (1) ask the model to generate an efficient version of code, and (2) ask the model to generate code and then directly optimize it (similar to the ALGO approach, despite not using test generation and execution to soften the environmental assumptions). Furthermore, the PIE [1] paper mentioned various few-shot prompting methods to perform code optimization, which should also be used as baselines. * To perform an "apple-to-apple" comparison over the models, Tables 1-4 should be better aligned. To understand which model is better at generating efficient code, they must be compared under the same set of tasks. While due to correctness inconsistency, the attributed coding tasks are not aligned, authors may compare the models under a set of mutually passing coding tasks. * Efficiency evaluation can be noisy without carefully configuring the metrics and testbeds. The considered metrics such as execution time and memory usage are flaky and non-reproducible as they are measured in physical platforms. To mitigate these limitations, authors may follow PIE [1] to use architecture simulators or other better-defined metrics. * While the author highlighted efficiency is critical in resource-constrained scenarios, the proposed technique is specialized for Python. It is unclear if Self-Optimization can work for performance-demanding languages such as C/C++ and related experiments are ignored. * [1] is a closely relevant and published work that should be well discussed and compared. [1] Shypula A, Madaan A, Zeng Y, Alon U, Gardner J, Hashemi M, Neubig G, Ranganathan P, Bastani O, Yazdanbakhsh A. “Learning Performance-Improving Code Edits.” ICLR 2024.
Questions
* How is SOAP compared to more reasonable baselines below: 1. Prompting an LLM to generate efficient code in zero-shot 2. Ask the LLM to refine its first generation for code performance 3. Fewshot prompting mentioned in PIE * Are profilers necessary? What if we just replace profilers with self-reasoning or self-reflection? * How's the cost (in the form of token consumption and pipeline running time) of SOAP and baselines? * How is the overall variation of mentioned metrics including execution time? For example, when running and measuring the code execution for N times, what is the overall coefficient of variation?
Rating
4
Confidence
5
Soundness
1
Presentation
4
Contribution
2
Limitations
* Using a profiler and code executor in the pipeline is expensive with respect to the degree of improvement. In production, integrating code execution in a sandbox is already challenging, and integrating a profiler (for multiple languages) would take more engineering cost. Meanwhile, the performance gain (e.g., execution time) overall looks minimal compared to the baseline whose prompt does not even encourage code optimization. * Overall the technical contribution of the paper is incremental: compared to prior techniques such as ALGO and PIE, the major difference is the use of code profiler.
Clarification on rejection
Thanks for the reply. I am sorry for not explaining my decision enough and here are my detailed reasons for my suggestion of decision: 1. First, the preliminary experimental results from the authors quickly cleared my concerns about approach effectiveness regarding weaknesses in #1, #4, and #5. 2. I won't reject the paper just for #1, #4, and #5 because those results are complementary to what the paper already had, meaning that they could be easily added to the camera-ready version if we end up accepting it. 3. Yet, weakness #2 still holds as existing results of the paper are mostly not represented in an "apple-to-apple" fashion. It is not even right to compare the efficiency of LLM codegen over different sets of tasks, making the efficiency score unreasonable to compare. For an extreme case, two LLMs may solve completely different sets of solutions, and computing efficiency scores over two totally different task sets cannot lead to clean and reasonable results regarding code efficiency in general. 4. Because it is problematic to compare LLM efficiency over different sets of tasks, ideally the author should globally correct the evaluation methodology, i.e., removing results made by inconsistent comparison and enforcing correctness alignments. This is crucial before accepting this paper because if not future work might inherit such a misevaluation methodology, leading to more inconsistent comparison in general. 5. Yet, doing reason 4 in rebuttal is quite hard objectively speaking. Meanwhile, given it's a major revision, it might be more rigorous to apply and reorganize the experiments and resubmit a clean version for another review, rather than accepting the paper through some rough views over the set of preliminary results that we are unsure if those can be systematically adopted in the camera-ready. To summarize, I deeply appreciate the efforts of the authors in the rebuttal; however, for soundness, the required revision goes beyond "convincing the reviewers through proof-of-concept experiments" that it is crucial to globally correct the evaluation methodology for positively leading future research in this thread. Last but not least, I strongly feel this work can be accepted after doing a major revision.
Apple-to-Apple results
Thank you for your reply. SOAP is effective for both scenarios (i.e., improving LLMs' own tasks and improving common correct tasks). To address the concern of the "apple-to-apple" results of our experiments in the Paper's main flow, in this thread, we provide results based on common tasks for Table 2 - Table 4's results, which provide consistent results for the two configurations. (*Table 1's results are already demonstrated in a previous thread.*) For Paper Table 2, where we discuss how SOAP's effectiveness would be affected by the optimization results, we provide the evaluation results in **Reviewer pc3i Table 2 (common tasks)**. First, we find the tasks addressed by both CodeLlama-70B-Ins and GPT-3.5-turbo-0301 with the Initial version (Step 0) and SOAP optimization process (Step 1-5). Then, we collect 46 tasks and measure the efficiency of LLM-generated code in these tasks. The evaluation results in **Reviewer pc3i Table 2 (common tasks)** show that CodeLlama-70B-Ins achieves SOAP efficiency results with one-step SOAP optimization, while GPT-3.5-turbo-0301 continuously improves the efficiency for the evaluated 46 tasks from Step 0 to Step 3. Then, the efficiency of GPT-3.5-turbo-0301 maintains efficiency for the other two optimization steps. Next, for Paper Table 3, where we discuss how different prompt optimization steps affect the efficiency of LLM-generated code, we provide the evaluation results in **Reviewer pc3i Table 3 (common tasks)**. First, we find the tasks addressed by both CodeLlama-70B-Ins and GPT-3.5-turbo-0301 for six different prompts and then get 40 tasks. Then, we provide the efficiency results in **Reviewer pc3i Table 3 (common tasks)**, where we can observe that SOAP also achieves SOTA efficiency results compared with other baselines. For example, SOAP achieves 3.72 (Mb*s) for TMU, while baselines only achieve 3.84 (Mb*s) in the TMU metric for GPT-3.5-turbo generated code. Finally, for Paper Table 4, where we discuss the differences in the efficiency of SOAP and the initial version of the CodeLlama family, we provide the evaluation results in **Reviewer pc3i Table 4 (common tasks)**. First, we detect correct tasks addressed by both SOAP and the Initial version for four CodeLlama models and then get 3 tasks. Next, we provide the efficiency results of CodeLlama family-generated code in **Reviewer pc3i Table 4 (common tasks)**. We can observe that for four CodeLlama models, SOAP continuously improves the efficiency of LLM-generated code. We hope that our newly provided results can address the reviewer's concern. *Note: We provide the Tables in the next thread.*
Results
***Reviewer pc3i Table 2 (common tasks)** Effect of the number of self-optimization steps in SOAP (46 tasks). The evaluation results are used to replace the results of Paper Table 2 with the same correct tasks addressed by two LLMs for all steps.* | Steps | **ET (s)** | **NET** | **MU (Mb)** | **NMU** | **TMU (Mb*s)** | **NTMU** | |-------|------------|---------|-------------|---------|-------|----------| ||| | | | || | **CodeLlama-70b-Instruct-hf** || | | | || | 0| 0.40| 3.00 | 54.79| 1.76 | 20.05| 4.86| | 1| 0.29| 2.20 | 54.90| 1.77 | 14.58| 3.53| | 2| 0.29| 2.20 | 54.90| 1.77 | 14.58| 3.53| | 3| 0.29| 2.20 | 54.90| 1.77 | 14.58| 3.53| | 4| 0.29| 2.20 | 54.90| 1.77 | 14.58| 3.53| | 5| 0.29| 2.20 | 54.90| 1.77 | 14.58| 3.53| ||| | | | || | **gpt-3.5-turbo-0301** || | | | || | 0| 0.24| 1.78 | 31.12| 1.00 | 10.74| 2.60| | 1| 0.22| 1.66 | 31.28| 1.01 | 7.36 | 1.78| | 2| 0.22| 1.64 | 31.27| 1.01 | 7.02 | 1.70| | 3| 0.22| 1.64 | 31.27| 1.01 | 6.98 | 1.69| | 4| 0.22| 1.64 | 31.27| 1.01 | 6.98 | 1.69| | 5| 0.22| 1.64 | 31.27| 1.01 | 6.98 | 1.69| ***Reviewer pc3i Table 3 (common tasks)** Effect of the number of self-optimization steps in SOAP (40 tasks). The evaluation results are used to replace the results of Paper Table 3 with the same correct tasks addressed by two LLMs for all steps.* | Steps| ET (s) | NET| MU (Mb) | NMU | TMU (Mb*s) | NTMU | | ------------------------- | ------ | ----- | ------- | --- | ---------- | ---- | | **CodeLlama-70b-Instruct-hf** | || | | || | Initial | 1.82| 14.78 | 25.85| 1.00| 36.32| 16.89| | Unsupervised Self-Refine | 0.34| 2.80 | 25.82| 1.00| 6.99| 3.25 | | Result-Aware Self-Refine | 0.33| 2.67 | 25.96| 1.00| 6.85| 3.19 | | memory_profiler | 0.26| 2.08 | 25.88| 1.00| 4.38| 1.97 | | time_overhead_profiler | 0.19| 1.57 | 25.88| 1.00| 4.03| 1.89 | | SOAP | 0.19| 1.55 | 25.87| 1.00| 3.98| 1.85 | | **gpt-3.5-turbo-0301** | || | | || | Initial | 0.24| 1.95 | 25.91| 1.00| 5.01| 2.33 | | Unsupervised Self-Refine | 0.26| 2.14 | 29.99| 1.16| 7.65| 3.56 | | Result-Aware Self-Refine | 0.23| 1.87 | 25.95| 1.00| 5.12| 2.38 | | memory_profiler | 0.19| 1.53 | 25.88| 1.00| 3.84| 1.79 | | time_overhead_profiler | 0.18| 1.47 | 31.26| 1.21| 4.98| 2.32 | | SOAP | 0.18| 1.48 | 25.88| 1.00| 3.72| 1.73 | ***Reviewer pc3i Table 4 (common tasks)** Effect of the number of self-optimization steps in SOAP (3 tasks) for CodeLlama family.* | Steps | **ET (s)** | **NET** | **MU (Mb)** | **NMU** | **TMU (Mb*s)** | **NTMU** | |-------|------------|---------|-------------|---------|-------|-----| || ||||| | | **CodeLlama-7b-Instruct-hf** | ||||| | | Initial | 0.78 | 5.61 | 30.82 | 1.00 | 13.89 | 5.50 | | SOAP (step 5) | 0.23 | 1.65 | 54.69 | 1.77 | 7.80 | 3.09 | || ||||| | | **CodeLlama-13b-Instruct-hf** | ||||| | | Initial | 0.33 | 2.38 | 54.66 | 1.76 | 11.44 | 4.53 | | SOAP (step 5) | 0.25 | 1.76 | 54.69 | 1.77 | 8.46 | 3.35 | || ||||| | | **CodeLlama-34b-Instruct-hf** | ||||| | | Initial | 0.31 | 2.25 | 54.59 | 1.76 | 10.72 | 4.25 | | SOAP (step 5) | 0.25 | 1.76 | 54.64 | 1.76 | 8.42 | 3.33 | || ||||| | | **CodeLlama-70b-Instruct-hf** | ||||| | | Initial | 0.32 | 2.31 | 54.62 | 1.76 | 10.97 | 4.34 | | SOAP (step 5) | 0.23 | 1.61 | 54.71 | 1.77 | 7.65 | 3.03 |
Friendly Reminder: Pending Reviewer Responses and Assessment Consideration
Dear Reviewer pc3i, Thank you for your review and comments on our paper, which you provided on August 12th. To address your concerns regarding the "apples to apples" comparison, we have added all experiments in the last response thread with the same format for our paper. We believe this will facilitate a more accurate and fair evaluation of our work. We would greatly appreciate it if you could take the time to review these results and consider improving your overall assessment of our paper based on these revisions. Thank you once again for your valuable feedback and for considering our request.
Summary
This paper studies an important and timely issue: the inefficiency often found in code generated by current Large Language Models (LLMs), which can result in longer execution times and higher memory consumption. To mitigate this issue, the paper proposes a self-optimization framework SOAP, designed to improve the efficiency of LLM-generated code. SOAP starts from the code generated by an LLM and then executes it locally to profile its execution time and memory usage. These profiles are fed back to the LLM, allowing it to revise the code and reduce overhead iteratively. The authors verify the effectiveness of SOAP through empirical experiments on the leading open-source and closed-source models, showing SOAP can substantially optimize both execution time and memory consumption.
Strengths
1) This paper proposes a novel general framework to optimize the efficiency of code generated by LLMs. It utilizes execution overhead profiles for self-diagnosis and iteratively enhances code efficiency. 2) Empirical experiments conducted on various leading LLMs demonstrate that this framework effectively optimizes both the execution time and memory consumption of LLM-generated code. 3) The framework does not require additional model training or tuning, making it flexible enough to be attached to arbitrary LLMs as a standalone pipeline. 4) The paper is well-written; the methodology introduced is concise, and the experimental results are robust and clear.
Weaknesses
1) The multi-iteration optimization method requires large token overhead, which can be both resource-intensive and time-consuming. 2) Overhead profiles serve as a good indicator for local code optimization. Nevertheless, the framework may struggle with the scalability of extended code global optimization due to the limited context window of LLMs.
Questions
1) In this paper, SOAP optimizes both execution time and memory usage simultaneously. However, in real-world scenarios, people might prioritize a single objective, either time or space. Do you think it is possible for SOAP to include optimization options (similar to -O3 or -Oz in GCC) to cater to different preferences? 2) The iterative self-optimization process might increase the risk that the final generated code is either non-executable or deviates from the original problem specifications. Do you have any mechanisms in place to prevent such deterioration? 3) How does the token usage in the SOAP profile optimization process compare to that in direct code generation (one-shot)?
Rating
8
Confidence
5
Soundness
3
Presentation
4
Contribution
4
Limitations
The authors addressed the limitations and potential negative societal impact of their work.
Summary
The paper introduces a novel method for code super-optimization by iteratively refining code using LLMs with profiling information. The focus is on enhancing the efficiency of LLM-generated code, addressing execution time and memory requirements. The proposed methodology involves generating code with an LLM and then iteratively feeding profiling results back to the LLM to optimize the code's efficiency. The profiling information includes per-line execution time and memory usage data, helping to identify specific lines that require significant overhead. Experimental results show that this approach outperforms existing unsupervised self-refine and result-aware self-refine methods on efficiency benchmarks like EffiBench, with supplementary experiments on HumanEval and MBPP.
Strengths
- The paper addresses a gap in current LLM code generation research by focusing on code efficiency. - The proposed iterative refinement method with profiling information is still innovative and provides a clear path to optimizing execution time and memory usage. - Experimental results demonstrate that the method significantly outperforms existing approaches in terms of code efficiency, showing the potential practical value of this technique.
Weaknesses
- LLM self-refinement or self-repair techniques have been extensively employed in generating responses/codes [1] and debugging tasks [2], whereas utilizing code profiling information for optimization is also a long-established concept [3, 4], which raises concerns in the proposed method's technical novelty. - The paper seems to treat the number of optimization steps as a hyperparameter. While Section 4.2 covers the impact of self-optimization steps, the average number of iterations required to achieve the optimization results in Table 1 is not clearly disclosed. This information is critical for comprehending the efficiency and practicality of the proposed method. Automating the stopping criteria for iterative optimization is also important in practice, but it is not supported by the proposed method, thus limiting its applicability. - The potential semantic degradation in code correctness after refinement, particularly in scenarios where test cases do not cover all edge cases (which are common in real-world applications), raises concerns about the reliability of the optimized code, particularly in safety-critical scenarios. - The methodology section, particularly section 3.4, is overly verbose. The explanation of each part of the prompt could be condensed to improve readability.
Questions
## Questions & Suggestions: 1. What is the average number of iterations required for optimization? What are the stopping criteria for iterative optimization? Did you set it to 5 for all the experiments? How is it determined when to stop the refinement process? 2. How significant is the profiling overhead? Is there any analysis on the average number of extra tokens required for including the profiling information as prompts? How does this affect the overall efficiency and effectiveness of the optimization process? 3. The downgraded pass@1 results in Section 4.4 suggest that the proposed method cannot guarantee semantic equivalence after code refinement. This could impact the practicality and reliability of the code superoptimizer, especially in safety-critical applications. How can this issue be mitigated? 4. Analysis of the optimization techniques applied by the LLM is necessary. Are there certain types of optimization where the method may fall short? 5. The current approach relies on the LLM to analyze and identify hot spots from the profiling results. Given the standardized format of profiling results, could rule-based post-processing techniques that filter out hot spot information improve or accelerate the self-refinement process? 6. In line 189, only code that passed all test cases is considered in the evaluation. What if the refined code fails the test cases during optimization? How is this situation handled? 7. In Table 2, the optimization effect on memory usage appears limited compared to execution time. Also, in the ablation studies (Table 3), self-optimization with only execution time profiling feedback achieves significant memory usage reduction. Does this mean that memory optimization is relatively easier in the evaluated benchmark, or does execution time information play a more critical role in optimization? --- [1] Madaan, Aman, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon et al. "Self-refine: Iterative refinement with self-feedback." Advances in Neural Information Processing Systems 36 (2024). [2] Olausson, Theo X., Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. "Is Self-Repair a Silver Bullet for Code Generation?." In The Twelfth International Conference on Learning Representations. 2023. [3] Chang, Pohua P., Scott A. Mahlke, and Wen‐Mei W. Hwu. "Using profile information to assist classic code optimizations." Software: Practice and Experience 21, no. 12 (1991): 1301-1321. [4] Agakov, Felix, Edwin Bonilla, John Cavazos, Björn Franke, Grigori Fursin, Michael FP O'Boyle, John Thomson, Marc Toussaint, and Christopher KI Williams. "Using machine learning to focus iterative optimization." In International Symposium on Code Generation and Optimization (CGO'06), pp. 11-pp. IEEE, 2006.
Rating
6
Confidence
4
Soundness
3
Presentation
2
Contribution
3
Limitations
The limitations related to the overhead of profiling and the semantic correctness of the optimized code are mentioned in the paper but not analyzed in detail. A more comprehensive cost-benefit analysis, including the extra tokens required for profiling information in prompts, would provide a clearer picture of the method's overall efficiency. The authors should also consider the broader implications of this optimization approach, including its potential impact on code reliability and safety in practical applications.
Response Reviewer 5c1j follow up questions
Thank you for your response to our rebuttal. We have conducted additional experiments and provided further clarification to address the concerns raised in the follow-up questions. We hope that these efforts have adequately addressed Reviewer 5c1j's concerns. If Reviewer 5c1j thinks that the issues have been satisfactorily resolved, we would greatly appreciate it if they could consider improving the overall assessment for SOAP, as the current average overall assessment is in a borderline scenario. **Q1** To address your concern, we first optimized the code based on the steps outlined in Paper Table 2 and then continued to optimize for an additional five steps to analyze the efficiency of LLM-generated code after a total of 10 optimization steps. Our evaluation results are shown below. We can observe that increasing the number of optimization steps decreases the overhead, but after five steps, the efficiency metric improvements are minimal. Therefore, in our paper, we mainly set the number of steps to 5. However, if developers want to optimize their code further, they can increase the optimization steps at the cost of some additional token usage. We believe this token usage is worthwhile, as the optimized code would be deployed in the software and run thousands to millions of times. | Steps | ET (s) | NET | MU (Mb) | NMU | TMU (Mb*s) | NTMU | |-------|--------|-----|---------|-----|------------|------| | CodeLlama-70B | | | | | | | | 0 | 0.36 | 2.50 | 91.25 | 2.45 | 157.50 | 19.75 | | 1 | 0.33 | 2.35 | 36.09 | 0.99 | 13.70 | 1.81 | | 2 | 0.31 | 2.18 | 36.09 | 0.99 | 13.04 | 1.72 | | 3 | 0.29 | 2.06 | 36.08 | 0.99 | 12.57 | 1.66 | | 4 | 0.29 | 2.03 | 36.08 | 0.99 | 12.50 | 1.65 | | 5 | 0.28 | 2.01 | 36.08 | 0.99 | 12.43 | 1.64 | | 6 | 0.27 | 2.00 | 36.07 | 0.99 | 12.40 | 1.63 | | 7 | 0.27 | 1.98 | 36.06 | 0.99 | 12.37 | 1.62 | | 8 | 0.27 | 1.98 | 36.05 | 0.99 | 12.35 | 1.61 | | 9 | 0.27 | 1.97 | 36.04 | 0.99 | 12.35 | 1.60 | | 10 | 0.26 | 1.96 | 36.03 | 0.99 | 12.35 | 1.59 | | GPT-3.5-Turbo-0301 | | | | | | | | 0 | 0.36 | 2.50 | 91.25 | 2.45 | 157.50 | 19.75 | | 1 | 0.33 | 2.35 | 36.09 | 0.99 | 13.70 | 1.81 | | 2 | 0.31 | 2.18 | 36.09 | 0.99 | 13.04 | 1.72 | | 3 | 0.29 | 2.06 | 36.08 | 0.99 | 12.57 | 1.66 | | 4 | 0.29 | 2.03 | 36.08 | 0.99 | 12.50 | 1.65 | | 5 | 0.28 | 2.01 | 36.08 | 0.99 | 12.43 | 1.64 | | 6 | 0.28 | 2.00 | 36.07 | 0.99 | 12.40 | 1.63 | | 7 | 0.27 | 1.99 | 36.06 | 0.99 | 12.38 | 1.62 | | 8 | 0.27 | 1.98 | 36.05 | 0.99 | 12.36 | 1.61 | | 9 | 0.27 | 1.97 | 36.04 | 0.99 | 12.35 | 1.60 | | 10 | 0.27 | 1.97 | 36.04 | 0.99 | 12.33 | 1.59 | **Q2** Response: **The difference between Initial pass@1 v.s. SOAP pass@1 is only the SOAP-generated incorrect code.** While the address method discussed in Q6 does not reflect in W3 & Q3, which means that with our solution in Q6, the incorrect code generated by SOAP in SOAP pass@1 will be replaced by the Initial generated code. **Q3.** We understand your concern about filtering out failed code in our evaluation. However, our approach is consistent with existing works in this field [1-6]. The primary reason for focusing only on correct code is that incorrect code may cause test cases to fail and terminate the execution process prematurely. Including such cases would lead to inaccurate results when measuring the efficiency of LLM-generated code. [1] Huang, Dong, Jie M. Zhang, Yuhao Qing, and Heming Cui. "EffiBench: Benchmarking the Efficiency of Automatically Generated Code." arXiv preprint arXiv:2402.02037 (2024). [2] Qiu, Ruizhong, Weiliang Will Zeng, Hanghang Tong, James Ezick, and Christopher Lott. "How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark." arXiv preprint arXiv:2406.06647 (2024). [3] Shi, Jieke, Zhou Yang, and David Lo. "Efficient and Green Large Language Models for Software Engineering: Vision and the Road Ahead." arXiv preprint arXiv:2404.04566 (2024). [4] Zheng, Jiasheng, Boxi Cao, Zhengzhao Ma, Ruotong Pan, Hongyu Lin, Yaojie Lu, Xianpei Han, and Le Sun. "Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Models." arXiv preprint arXiv:2407.11470 (2024). [5] Waghjale, Siddhant, Vishruth Veerendranath, Zora Zhiruo Wang, and Daniel Fried. "ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?." arXiv preprint arXiv:2407.14044 (2024). [6] Du, Mingzhe, Anh Tuan Luu, Bin Ji, and See-Kiong Ng. "Mercury: An efficiency benchmark for llm code synthesis." arXiv preprint arXiv:2402.07844 (2024).
Friendly Reminder: Pending Reviewer Responses and Assessment Consideration
Dear Reviewer 5c1j, Thank you for your review and comments on our paper on 13 Aug, we have provided the clarification and other experiment results to address your concern. Since the discussion period will be closed in the next six hours, while we do not receive the message about whether we have addressed all of your concerns, we would greatly appreciate it if you could take the time to review these results and consider improving your overall assessment of our paper based on these revisions. Thank you once again for your valuable feedback and for considering our request.
Summary
This paper presents SOAP, a new code generation approach that improves the efficiency of code generated by a LLM. SOAP adopts a self-refinement method that iteratively prompts the LLM to re-generate the code based on the profiling results. Specifically, it uses the line-profiler package in Pyhton to get the execution time profiling results and the memory_profiler package to get the memory usage profiling result. SOAP is evaluated on three benchmarks and 22 LLMs, including both 16 open-sourced and 6 closed-sourced models. The results show that SOAP can effectively improve the execution time and memory usage of code generated by most LLMs. While it slightly decreases the LLM's performance on functional correctness, the impact is small.
Strengths
1. SOAP is evaluated on multiple benchmarks and many LLMs. 2. The results look promising. 3. The paper is well-written and easy to follow.
Weaknesses
1. The technical novelty of this work is limited. The self-refinement method proposed in this work is very similar to existing self-refinement methods such as Self-Edit and Critic. The only difference is that SOAP uses profiling results while existing methods use testing results, etc. 2. The authors ignored existing work on using transformers or LLMs for code optimization. They should discuss these papers in the related work and also use them as comparison baselines. - Liu, Shengcai, et al. "Large language models as evolutionary optimizers." arXiv preprint arXiv:2310.19046 (2023). - Chen, Zimin, Sen Fang, and Martin Monperrus. "Supersonic: Learning to generate source code optimisations in c/c++." arXiv preprint arXiv:2309.14846 (2023). - Shypula, Alexander, et al. "Learning performance-improving code edits." arXiv preprint arXiv:2302.07867 (2023). 3. Section 4.3 compares SOAP with Self-Refine and Reflexion. It is unclear how Self-Refine and Reflexion were configured in this experiment. Did the authors use the original prompts from Self-Refine and Reflexion? Or did the authors modify their prompts to include instructions on code efficiency? If the former, there is a comparison fairness issue since the original prompts in Self-Refine and Reflexion are not designed for code optimization. If the latter, the authors should provide the prompts used in Self-Refine and Reflexion and justify the prompt design. 4. The experiments on the impact of step-optimization steps (Section 4.2) only include two big models. The findings may not be generalizable to smaller models. Why not also include some smaller models? 5. The experiments on the impact of functional correctness were only done on EffiBench. Since EffiBench is not specifically designed for functional code generation, the results may not be representative. The authors should also experiment with HumanEval and MBPP and report the pass@1 scores on these two benchmarks.
Questions
1. What is the key technical novelty of this work? 2. How would you compare SOAP with existing transformer or LLM-based code optimization work? 3. What are the prompts used in the experiments with Self-Refine and Reflexion?
Rating
3
Confidence
5
Soundness
2
Presentation
3
Contribution
2
Limitations
The discussion on limitations looks reasonable.
Friendly Reminder: Pending Reviewer Responses and Assessment Consideration
Dear Reviewer H3VZ, Thank you for your review and comments on our paper. Since the discussion period will be closed in the next six hours, while we do not receive the message about whether we have addressed all of your concerns, we would greatly appreciate it if you could take the time to review these results and consider improving your overall assessment of our paper based on these revisions. Thank you once again for your valuable feedback and for considering our request.
Response to the rebuttal
Thank you for the responses, and sorry for the late response due to a flurry of proposal and review dues. Regarding the novelty, would it be fair to say the essential difference to self-refine is the inclusion of the profiling results from the `line_profiler` library in the prompt? While I appreciate the authors' effort in conducting new experiments quickly in the short rebuttal period, I feel these results require another round of careful reviews, especially given that some results look quite surprising and counterintuitive. Without knowing more details and experiment settings, it is hard to make a full assessment of the correctness of the results. In particular, it's surprising to see that the pass@1 rates of all these models barely change after applying SOAP. Note that compared with the original code generation prompt, SOAP adds a lot of line-level profiling results, which significantly deviates from the original prompt. Presumably, we should see some deviations in pass@1, as LLMs are pretty sensitive to prompt design and including many profiling results may distract the LLM from reading the original task description. How did the authors keep the pass@1 rates almost unchanged? Furthermore, the evaluation results of PIE with CodeLlama7B look quite strange. The average execution time of code optimized by PIE is in the range of 0.4s to 0.9s. However, the execution time of CodeLlama7B in Table 1 is 4.7s before SOAP optimization and 4.52s after SOAP optimization. If the execution time after PIE optimization is correct, **does this mean PIE can achieve 10x speed up compared with the initial code and the code optimized by SOAP alone?** If that's the case, what's the point of applying SOAP on top of PIE given that PIE has already achieved such a big improvement? Overall, I feel this paper requires another round of careful reviews before it can be accepted.
Response to Reviewer H3VZ
Dear Reviewer H3VZ, Thanks for your response before the discussion period deadline, which allows us to further clarify the potential misunderstanding and concerns that would affect your overall assessment. First, compared with self-refine, SOAP utilizes line_profiler and memory profiler to catch the execution time and memory usage of each line in the LLM-generated code. Second, for the pass@1 of LLM-generated code of pre-SOAP and post-SOAP, we want to mention that in https://anonymous.4open.science/r/SOAP-FF6C/src/SOAP.py (Lines 177-Lines 185), we provide tolerance for SOAP that requires the SOAP optimized code can pass public test cases and then replace original code. If the optimized code can not pass the public test cases, SOAP optimized code will not replace the original code, and we will use the original code for the next step of optimization (Lines 184-185). In this step, the coverage of the public test cases would largely affect the correctness of the final SOAP-optimized code. After the empirical study, we observe that the code line coverage of EffiBench for public test cases achieves **99.24%**, which makes sure that the pass@1 after the self-optimization process (SOAP) only has a few decreases, and even in some models, the pass@1 would not decrease. Third, we need to mention that PIE already trained CodeLlama with its provided code, which means that the pass@1 of PIE+CodeLlama-7B may be different, and the tasks addressed by PIE and the original CodeLlama are different. Then, some tasks with higher execution time (e.g., 4.7s) may not addressed by PIE-CodeLlama, and the default (initial) ET and other metrics are different. We hope that Reviewer H3VZ can understand this difference between the original CodeLlama and PIE-CodeLlama. We hope that our clarification can address Reviewer H3VZ's concern about **I feel these results require another round of careful reviews, especially given that some results look quite surprising and counterintuitive.**
Author-Reviewer discussions (Aug 7 - Aug 13)
Dear Submission5230 reviewers, We appreciate your reviews as we move into the Author-Reviewer discussions phase (Aug 7 - Aug 13). Please read all reviews and author responses carefully. Please address any remaining questions or concerns with the authors and respond to their rebuttal as needed. Authors need time to respond to your messages, so please post your responses as soon as possible, so there is time for back-and-forth discussion with the authors. At a minimum, please acknowledge that you have read the author rebuttal. Based on the material provided, please adjust your scores and reviews as necessary. Dear Submission5230 authors, We appreciate your rebuttal. Please continue the dialogue with the reviewers during this discussion phase. This message and thread are visible to both reviewers and authors. If you need to respond privately, adjust the "Readers" setting accordingly.
Question for authors
I appreciate your submission, detailed reviewer reviews, and your extensive rebuttal. One question that I did not see discussed yet that might be important for readers and reviewers: In Table 1, large percentage improvement is mostly achieved for very inefficient code especially for memory metrics. (StarCoder2-15B ET is impressive exception). You have a number of code solutions of various efficiency. It would be interesting to see the performance of SOAP for the code (not necessarily generated by the same LLM) that ranges from bad to "optimal" (the best available) solution. Since the above would require a lot of experiments (N code variants x M LLMs), an alternative would be to provide ratio of best code metric after SOAP (irrespective of LLM) to best code before SOAP (irrespective of LLM). E.g. for ET in Table 1, this would be 0.12 / 0.29. This is cruder metric, since there is no guarantee that 0.12 can be achieved by running SOAP on 0.29 code, but it gives some measure of how much SOAP can improve good code. For example, for MU this metric shows very low improvement of 24.28 / 24.31.
Response about global optimal
Dear AC, Thanks for your efforts to remind reviewers to participate in the discussion process. To address the concern about the global efficiency results, we conduct experiments with our previously generated source code and SOAP-optimized code. We follow the below instructions to calculate the overall efficiency. First, we collect all correct LLM-generated code for each task and then select the most efficient code generated by LLMs based on ET, MU, and TMU. Then, we calculate the overall efficiency metrics for each metric-guided selected code for the initial code and SOAP-optimized code. The evaluation results for EffiBench are shown below. We can observe that in this scenario, SOAP largely improved the efficiency of LLM-generated code. For example, in the Optimizal ET results (i.e., we collect source code for both initial and SOAP optimized code for each task), SOAP decreases the average ET from 0.97 (s) to 0.23 (s). In Optimizal MU, SOAP decreases the MU from 59.16 (Mb) to 21.13 (Mb). In Optimial TMU, SOAP also decreases the TMU from 137.18 (Mb*s) to 10.14 (Mb*s). We hope our experiments can address your concern. | Steps | ET (s) | NET | MU (Mb) | NMU | TMU (Mb*s) | NTMU | |---------|--------|-------|---------|------|------------|------| | Optimal ET | | | | | | | | Initial | 0.97 | 7.55 | 59.26 | 1.75 | 153.39 | 27.01| | SOAP | 0.23 | 1.59 | 29.74 | 1.03 | 13.71 | 1.59 | | Optimal MU | | | | | | | | Initial | 1.33 | 10.43 | 59.16 | 1.75 | 169.51 | 29.85| | SOAP | 0.74 | 4.91 | 21.13 | 0.83 | 21.22 | 1.93 | | Optimal TMU | | | | | | | | Initial | 1.38 | 10.95 | 59.25 | 1.75 | 137.18 | 26.13| | SOAP | 0.47 | 3.41 | 31.17 | 1.16 | 10.14 | 1.37 |
Thank you for your response.
I appreciate your response. I think this is a good metric. However, could you clarify the numbers in your table? I would think that "Optimal ET" would have initial 0.29 and after-SOAP 0.25 (DeepSeek-33B-Ins and OpenCodeInterpreter-33B are most efficient based on ET pre-SOAP, I picked DeepSeek-33B-Ins since SOAP improves its code more), "Optimal MU" would have initial 24.31 and after-SOAP 24.28 (StarCoder2-3B is most efficient based on MU), "Optimal TMU" would have initial 13.06 and after-SOAP 11.54 (OpenCodeInterpreter-33B is most efficient based on TMU). Am I misunderstanding your metric? (I did not look up other columns in the table just the ones corresponding to the Optimal metric in your subheading) Thank you again.
Response for AC question
Dear Area Chair gHRy, Clarification for Previously Provided Results: Thank you for your response. We would like to address your concerns and clarify our setup as follows: In the experiment for optimal ET, MU, and TMU, we first gather solutions from all LLMs for each task in our EffiBench. Subsequently, we calculate the ET, MU, and TMU for each solution from the collected solutions and rank them based on these three metrics. For instance, considering ET, we select the most efficient solution (that is, a solution with the lowest ET among all model-generated codes) as the pre-SOAP solution. This represents the most effective solution in terms of ET in the pre-SOAP stage. Similarly, for SOAP, we follow the same strategy to obtain the most efficient ET solution. Through this process, we do not directly select the 0.29 (s) OpenCodeInterpreter-33B's pre-SOAP as the optimal ET since some tasks might not be addressed by OpenCodeInterpreter-33B. Other LLMs that may address these tasks are also used to calculate pre-SOAP. Lastly, we compute the efficiency for both the pre-SOAP most efficient code and the post-SOAP most efficient code, but only for correct codes in the experiments. Indeed, the method you proposed is excellent. Based on your suggestion, we have conducted the following experiments. As can be seen, in this setup, SOAP also reduces the overhead of the initial version across all metrics. For instance, the ET decreases by 58.6%, reducing the average execution time from 0.29 seconds to 0.12 seconds. In our final version, we will also provide the evaluation results of SOAP for the N code variants x M LLMs. | Steps | **ET (s)** | **NET** | **MU (Mb)** | **NMU** | **TMU (Mb*s)** | **NTMU** | |---------|------------|---------|-------------|---------|----------------|----------| | Initial | 0.29 | 1.25 | 24.31 | 0.93 | 11.84 | 1.11 | | SOAP | 0.12 (58.6\%)| 1.03 (17.6\%)| 24.28 (0.1\%)| 0.82 (11.8\%)| 2.03 (82.9\%)| 0.88 (20.7\%)|
Thanks for the detailed reply!
First, I'd like to thank the authors for their reply. It is impressive and surprising to add this volume of preliminary experiments within such a short period. While I think weaknesses #1, #4, and #5 can be mostly addressed by adding new tables in the paper, weaknesses #2 and #3 require a more thorough global revision/update. Specifically, while in the demonstrated comparison from the preliminary results SOAP outperforms other baselines, the ratio of improvements is largely different from table to table when using different measurements and task set alignments. For example, improvements in Table 3 are way larger than that of Table 2 and this is likely due to the impact of task correctness over efficiency. That being said, while I appreciate the various new preliminary experiments from the authors, I still want to hold a rejection before an optimal experimental setting is figured out and applied globally. This would largely improve conclusion consistency and soundness of evaluation, which is very crucial for efficiency evaluation. Regarding the responses to my questions, I have a few concerns: 1. The variations in Table 5 are surprisingly small and the variations across different metrics are surprisingly equivalent. Can the authors double-check their data? 2. How do the authors configure Result-Aware Self-Refine compared to SOAP? Follow-up suggestions: 1. Given the large token overhead of SOAP, what if we just do optimization sampling and profile to select the best one using the same token budget? 2. In general, it is encouraged to simplify the metrics for clarity.
Rebuttal for Reviewer 4 Follow Up
**Specifically, in the demonstrated comparison from the preliminary results SOAP outperforms other baselines, the ratio of improvements is largely different from table to table when using different measurements and task set alignments. For example, improvements in Table 3 are way larger than that of Table 2 and this is likely due to the impact of task correctness over efficiency. That being said, while I appreciate the various new preliminary experiments from the authors, I still want to hold a rejection before an optimal experimental setting is figured out and applied globally. This would largely improve the consistency and soundness of the conclusion, which is crucial for efficiency evaluation.** Response: Thank you for your appreciation of the new preliminary experiments we provided during the rebuttal process. However, we are disheartened that reviewer pc3i wishes to reject our paper based on the claim, **"I still want to hold a rejection before an optimal experimental setting is figured out and applied globally"**. To address reviewer pc3i's concerns about w2 and w3, we have provided new results to reviewer pc3i's requirements. Specifically, for w2, we want to clarify that we conducted experiments on **all** closed-source models. However, **we did not conduct experiments on open-source models due to their low pass@1 and the lack of consistent correct code across all open-source models**. Next, for the PIE provided Gem5 default simulator, we also included seven different LLMs: smaller open-source LLMs (OpenCodeInterPreter-1.3B, DeepSeekCoder-1.3B-Ins, CodeLlama-7B-Ins-hf, StarCoder2-15B), a larger open-source model (CodeLlama-70B-Ins-hf), and two closed-source models (GPT-3.5-turbo-0301 and Claude-3-sonnet). We believe that these models are representative of our experiments. **The variations in Table 5?** Response: To address the reviewer’s concern, we provide the source code in our anonymous GitHub. We first use run_source_code_five_times.py to run the LLM-generated code five times. Next, we use report_variant.py to first calculate mean and std. Then we report coef_of_var with (std_dev / mean) * 100 for all experiments. **Configuration of Result-Aware Self-Refine compared to SOAP?** Response: The configuration of Result-Aware Self-Refine is shown in our anonymous GitHub prompts.py, where the only difference between SOAP and Result-Aware Self-Refine is that the overhead analysis part of Result-Aware Self-Refine in Lines 19 only contains the ET, MU, and TMU (See Lines 54-58). The overhead analysis part of SOAP also contains the execution time profiler and memory profile (See Lines 28-35). **Follow-up suggestions:** **S1: Given the large token overhead of SOAP, what if we just do optimization sampling and profile to select the best one using the same token budget?** Response: Thank you for your suggestion. We have provided evaluation results for your suggested method and SOAP in GPT-3.5-turbo-0301 below. It's observable that this method can enhance the efficiency of code generated by the LLM as compared to the initial version. For instance, the TMU of GPT-3.5-turbo-0301 decreases from 157.50 (Mbs) to 27.99 (Mbs). This significant reduction is primarily due to the suggested method of selecting the most efficient route from multiple codes. However, we also noticed that the suggested method is still less efficient than SOAP. The primary reason for this is that the suggested method primarily aims to generate multiple efficient solutions for a given task. However, without the profile information provided by SOAP, the LLM is unable to optimize the code and generate a more efficient version than the initial code, resulting in the suggested method being less efficient than SOAP. | Strategy | **ET (s)** | **NET** | **MU (Mb)** | **NMU** | **TMU (Mb*s)** | **NTMU** | |----------|------------|---------|-------------|---------|----------------|----------| |Initial|0.36|2.50|91.25|2.45|157.50|19.75| | Reviewer pc3i suggested | 0.31 | 2.17 | 36.14 | 1.00 | 27.99 | 2.94 | | SOAP | 0.28 | 2.01 | 36.08 | 0.99 | 12.43 | 1.64 | *Effectiveness of SOAP with Reviewer pc3i proposed optimization sampling and profile to select the best one for GPT-3.5-turbo with similar token usages. For each method, we use **int(token usage of SOAP/token usage of baselines such as direct generation with sampling) + 1** to ensure that the token usage of SOAP should be lower than the baselines.* **S2: In general, it is encouraged to simplify the metrics for clarity.** Response: Thanks for your suggestion. We will revise our manuscript in the final version.
Thank you for your response
Thank you for your thorough responses. The additional experiments and explanations have satisfactorily addressed my concerns. As a result, I am pleased to elevate my confidence from 4 to 5 and advocate for the paper's acceptance. I also read comments from the Area Chair and other Reviewers. I like the idea of evaluating N code variants x M LLMs, as it promises to provide a comprehensive performance spectrum across various LLMs. However, it is clear that conducting such an intensive experiment within the rebuttal period is impractical. Could you share the individual model performances (or percentages) from your experiment in the AC thread? It would be intriguing to see the expertise demonstrated by each individual model.
Thanks for your reply. We provide the distribution below. We can observe that GPT-4-turbo-preview has the highest distribution in Pre-SOAP efficiency results. GPT-3.5-turbo-0301 achieves the highest efficiency ratio in Post-SOAP efficiency results. *Distribution of the most efficient code for each task of EffiBench. We provide each LLM ratio for the most efficient code and then report the ratio for the overall results.* | Model | Pre-SOAP (%)| Post-SOAP (%)| |-----------------------------------|----------|-----------| | OpenCodeInterpreter-DS-1.3B | 2.33 | 1.78 | | OpenCodeInterpreter-DS-6.7B | 4.66 | 2.88 | | OpenCodeInterpreter-DS-33B | 6.30 | 3.70 | | deepseek-coder-1.3b-instruct | 3.84 | 1.10 | | deepseek-coder-6.7b-instruct | 5.07 | 7.81 | | deepseek-coder-33b-instruct | 0.00 | 0.00 | | CodeLlama-7b-Instruct-hf | 1.23 | 0.55 | | CodeLlama-13b-Instruct-hf | 3.29 | 0.82 | | CodeLlama-34b-Instruct-hf | 4.52 | 1.51 | | CodeLlama-70b-Instruct-hf | 5.34 | 0.82 | | XwinCoder-13B | 4.11 | 0.55 | | XwinCoder-34B | 5.75 | 2.74 | | WizardCoder-Python-13B-V1.0-GPTQ | 1.10 | 0.41 | | starcoder2-3b | 0.68 | 1.23 | | starcoder2-7b | 1.37 | 0.41 | | starcoder2-15b | 0.55 | 0.14 | | gpt-3.5-turbo-0301 | 8.77 | 27.67 | | gpt-3.5-turbo-0613 | 7.53 | 5.07 | | gpt-3.5-turbo-1106 | 7.12 | 4.52 | | gpt-4 | 7.53 | 7.53 | | gpt-4-turbo-preview | 16.16 | 14.25 | | claude-3-haiku | 1.37 | 5.89 | | claude-3-sonnet | 1.37 | 8.63 |
Thank you for the detailed response and the supplementary experiment results. I have a few follow-up questions: 1. Could the optimization performance be further improved with steps greater than 5? Is there a way to determine convergence in self-refinement, especially considering that the number of iterations affects token usage and LLM inference costs (as in W2)? 2. In your response to Q6, you mentioned that the entire task is filtered out if the refined code fails any private test. Can I interpret the difference in your answer to W3 & Q3 (Initial pass@1 v.s. SOAP pass@1) as the proportion of filtered tasks? 3. I do not think it is reasonable to filter out failed code in your evaluation. In real-world scenarios, private cases are supposed to be invisible.
Decision
Accept (poster)