Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation

Summary

This paper leveraged large language model (LLM) based and mutation-based strategies to generate high-quality test cases for the popular dataset HumanEval. The extended dataset HumanEval+ provides a better code generation benchmark for assessing the performance of LLMs such as ChatGPT and GPT4. Experimental results showed that compared to the HumanEval dataset, the extended one can detect more issues of the generated code, where the generation performance of 19 LLMs is reduced by 13.6-15.3% on average in terms of pass@k. To save the testing time, this paper also provided a refined set of test cases with fewer numbers but with the same code coverage, killed mutants, and sample killings.

Strengths

+ Proposing a test case generation method for LLMs, which combines the LLM-based and mutation-based strategies. + Evaluating 19 LLMs on the extended datasets and showing the overestimation of the original dataset. + Providing a test-suite reduction for quick evaluation.

Weaknesses

- Some technical details, such as the quality of seed inputs, are unclear, which affect the soundness of the proposed approach. - There are some unclear claims/overclaims. - Lack of comparisons with other test case generation methods and related work. The claimed contribution indicates “that prior popular code synthesis evaluation results do not accurately reflect the true performance of LLMs for code synthesis”. Can the extended dataset proposed by this paper reflect the “true performance”? Similarly, the paper said that the existing tests “often fall short in capturing all possible scenarios”, leading to “false confidence” in the results. Can your test cases “capturing all possible scenarios”? Furthermore, this paper claimed that the proposed benchmark can “precisely evaluate the functional correctness of LLM-generated code” by generating “interesting test inputs”. It is unclear to me how “precise” can the evaluation provide and what are the “interesting test inputs”. Please clarify them. EvalPlus “first uses ChatGPT to generate a set of high-quality seed inputs for later mutation”. The quality of the seed inputs is not verified. Are these inputs correct? How do you ensure the quality of the generated seed inputs? and how many seed inputs did the ChatGPT generate? What are the prompts used? The type-aware input mutation “randomly” selected some inputs from the generated seed pool as the inputs of mutants. This pipeline did not clearly describe how randomness affects the result of the input mutation and the quality of the evaluation. The proposed approach “adopted a programming by contract philosophy by systematically adding code assertions as contracts (e.g., assert n > 0) to ensure the test inputs for the function are well-formed”. The context did not tell how to define the assertions specifically, how to ensure the correctness of the definitions, and which docstrings need clarification with these assertions. Will it incur a lot of manual effort? For the evaluation, the paper did not compare the method with other baseline methods. The paper says that traditional automated test generation methods are inapplicable to generating semantically meaningful inputs. Why is that so? More descriptions are needed here. Otherwise, it is difficult to determine the advantages of the proposed method over the traditional methods. Recently, there is also a related work on applying ChatGPT prompts to code generation, the authors also discussed the quality/correctness of the generated code: C. Liu et al., Improving ChatGPT Prompt for Code Generation, https://arxiv.org/abs/2305.08360.

Questions

How is the quality of the seed inputs generated by ChatGPT? Are these inputs correct? How do you ensure the quality of the generated seed inputs? and how many seed inputs did the ChatGPT generate? What are the prompts used? Will the quality of seed inputs affect your approach significantly?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors addressed the limitations in the Appendix.

Summary

In this paper, the authors introduce EvalPlus, an evaluation framework crafted to assess the code generation of LLMs. Combining LLM and mutation-based techniques, EvalPlus diversifies the generated test inputs, thereby broadening the evaluation spectrum for LLM-produced code. Through comprehensive experimentation, the authors highlight EvalPlus's ability to expose shortcomings in previous LLM-based code evaluations. Additionally, the study brings HumanEval Plus, an enriched dataset built upon the existing HumanEval, offering an abundance of test inputs and a more dependable ground truth for improved reliability.

Strengths

* This paper distinguishes itself through its innovative approach to test input generation. It extends beyond the conventional usage of ChatGPT for creating test inputs, by incorporating a test-mutation technique and a “distill” method. This results in a diverse yet non-redundant range of test inputs, illustrating the authors' commitment to crafting a robust and efficient evaluation framework. * This paper is well-written and easy to read. * The study stands out for its well-designed experiment, which encompasses a comprehensive selection of contemporary LLMs.

Weaknesses

* HumanEval [8], in comparison to real-world coding projects, presents a significantly simpler challenge. It's a benchmark that's even simpler than others, such as APPS [Hendrycks et al.]. Given this context, the effectiveness of EvalPlus when applied to more complex tasks remains uncertain. This indicates a valuable area for enhancement in this study—specifically, future research could delve into assessing how well EvalPlus performs in more complicated, real-world coding scenarios. Such explorations could contribute crucial insights to understanding the scalability and adaptability of EvalPlus.

Questions

* Given the impressive results demonstrated by EvalPlus within the comparatively simpler context of the HumanEval benchmark, how do you envision expanding this work to more complex environments? Specifically, what strategies or modifications do you plan to implement to ensure EvalPlus is effective in assessing code generated for real-world projects, such as those found in open-source Python libraries? Could you elaborate on potential challenges you anticipate and how you intend to navigate them in this expansion?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

* While this study presents a promising step forward with EvalPlus, a limitation lies in the evaluation scenario chosen for testing. Given that the proposed method is intended to be general, using the relatively simple HumanEval dataset for testing may not sufficiently demonstrate the framework's generalization capabilities. To conclusively assert its wide applicability and robustness, it would be beneficial for future research to include tests on more complex datasets or in more challenging real-world contexts, such as open-source Python libraries. This would provide a more comprehensive understanding of EvalPlus's potential in diverse, practical applications.

Summary

The authors propose an evaluation framework for validating the correctness of large language model-generated code. In particular, the framework first utilizes ChatGPT to generate multiple seed inputs, which are then expanded into a large set of inputs through type-aware mutation. In addition, to ensure evaluation efficiency, the input set can be reduced by employing code coverage, mutant killings, and LLM sample killings.

Strengths

The proposed solution addresses the problem of insufficient testing for LLM-generated code. The expanded dataset can outperformance the original one, while the reduced dataset achieves similar results as the expanded one. The authors evaluate the proposed solution via comparative analysis of 19 large language models, analyze the pass rate distribution of the employed dataset HUMANEVAL, and identify several errors in the ground-truth solutions. The related work section discusses previous research on large language models for code, the coding benchmark, and automated test generation. The authors discuss the research gap between this study and the related work.

Weaknesses

The authors do not employ a more capable large language model in the proposed solution, and there is a lack of evaluation of efficiency.

Questions

- I wonder if GPT-4 can generate more interesting seed inputs than ChatGPT. Explicitly clarifying the employed component in EvalPlus may affect the generality of the solution, I suggest this can be introduced as the implementation or setup information. - The authors can elaborate on how to filter out the invalid inputs in Section 2.1. - The evaluation does not examine the efficiency of type-aware input mutation, and test-suite reduction. - Table 4: “Killed samples” has ambiguity. - Section 2: “EvalPlus obtains a augmented benchmark …” -> “an”.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors do not explicitly identify the potential societal impact or limitations of this work, but they claim that their future work include conducting testing to more code benchmarks.

Summary

This paper introduces a rigorous evaluation framework EvalPlus for program synthesis driven by automated test generation. For automated test generation, this work proposes to combine both LLM-generated tests and mutation-based input generation to largely augment the text inputs for an existing code benchmark of HumanEval. EvalPlus is evaluated with a diverse set of code LLMs and results show that the new augmented benchmark can identify a significant amount of previously undetected wrong code generated by LLMs, leading to a more accurate evaluation. It further introduces a mini-version benchmark reserving almost the same test effectiveness.

Strengths

* The paper studies how to evaluate code LLMs more accurately, which has become an important topic in the era of LLMs as coding is one of the key capabilities of LLMs to showcase. The proposed EvalPlus of combining both LLM-generated tests and mutation-based tests is well motivated and technically sound. * The paper is well written and easy to follow. The evaluation is very comprehensive (considering most of state-of-the-art LLMs) and provides convincing results to authenticate the effectiveness of the proposed EvalPlus.

Weaknesses

* The biggest weakness of this paper is the lack of discussion and comparison to other related work of AlphaCode and CodeT [1], which share common techniques such as mutation-based input generation and unit tests generated by LLMs. In this regard, the novelty of EvalPlus is relatively limited. Note that in AlphaCode, they have already explored similar mutation-based techniques to largely augment the test cases for program synthesis. Besides, they also explored training a LLM-based test input generation model to generate test cases in a clustering process. For CodeT, they have explored using LLMs to generate test cases, though these generated test cases are used for a different purpose of reranking the generated programs. The authors should discuss and compare with these works to better justify their novelties. [1] CODET: CODE GENERATION WITH GENERATED TESTS

Questions

* Can you explain more on the difference between EvalPlus and AlphaCode/CodeT? * I noticed that this work compares with the recently released StarCoder and CodeGen2, but not another SoTA code LLM of CodeT5+ which was released at the same time with better results on HumanEval. Any reasons for not including it?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I did not find any discussion on limitations from the paper.

Summary

This paper describes an enhanced test dataset and test-driven evaluation of code generation by LLMs. The paper compares different LLMs and a stricter metrics using the enhanced evaluation dataset and shows that these LLMs are about 15% less correct than is reported based on earlier test datasets.

Strengths

Given that LLMs appear to achieve a number of programming tasks but often fail to completely get them right. They fail in unexpected ways and places. Improving the evaluation metrics is critical to use of LLMs in product and commercial contexts. This paper improves the test beds by synthetically creating addtional tests through a framework that uses both LLMs and mutation based techniques. The paper demonstrates that about 13-15% of code generated by typical LLMs that would be qualified as passable according to previous metric is disqualified by this method. Since LLMs are often used for NL->code contexts this problem may arise because of inssufficient tests or because of less specific description in NL of the code to be written. They increase the test suites by 81X and also create a reduced version of it that shrinks it 47x for fast evaluation. The test suites are enhanced by augmenting the prompts with ground truth tests and generating higher quality seed data. From these seed inputs, type-aware mutations are created to enhance the test data sets.

Weaknesses

While the enhanced data set does call out more problems in the LLM-generated code, it is not clear if it is the best it can do. For instance, just by running a static analyser, a syntax checker or some other engine, the issues in the code generated can be found and fixed. The authors should have at least compared one such approach. There is no comparison of another approach that improves the performance of LLMs in context. One of the ways LLMs are often used for code generation is to generate good enough code and then through human interaction or through other static tools rerank, correct or qualify the generated code. The enhanved data set is diefinitely useful but not clear how useful in that it flags about 13-15% of the code. Some human evaluation or A/B test to qualtitatively say how good this is would have been more insightful.

Questions

1. Have you tried any A/B experiment or other human evaluation to see how much more useful is this improved test set to the LLM context in a real application? 2. Have you thought about other LLM tasks which have similar challenges. 3. As LLMs are constantly improving likely the value of such an enhanced test set diminishes. Have you thought about studying that?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Limitations - LLMs are used in context of human input. So it is not clear how useful this enhanced data is. It is also not clear if the enhanced test set increases coverage in criticial dimensions. LLMs are constantly improving and in the absence of a good A/B test or application context it is hard to tell the usefullness of the system.

Reviewers, please respond

Reviewers, please respond to the rebuttal. The authors state that the primary contribution is the: > generated dataset and accompanying rigorous evaluation study on recent popular LLMs, which demonstrates for the first time that prior popular benchmarks are insufficient to evaluate functional correctness of LLM generated programs. The main question for you is whether you think this is of sufficient interest for publication.