Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM

Summary

The paper introduces EvoEval, a novel benchmark suite for evaluating the coding abilities of large language models (LLMs) that addresses the limitations of existing benchmarks by evolving them into more varied and targeted domains. Through empirical analysis of 51 LLMs, the authors demonstrate that LLMs perform significantly worse on EvoEval compared to traditional benchmarks like HumanEval, with an average performance drop of 39.4%. This decrease, ranging from 19.6% to 47.7%, suggests potential overfitting to current benchmarks and results in significant changes in LLM rankings. Additionally, the paper highlights the brittleness of LLMs to changes in problem wording and underscores the importance of problem composition skills in programming tasks. EvoEval not only broadens the scope of evaluation but also provides a framework for continually adapting benchmarks in response to advancements in LLM capabilities.

Rating

8

Confidence

5

Ethics flag

1

Reasons to accept

1. The paper addresses a critical need by highlighting that existing code generation benchmarks, such as HumanEval and MBPP, suffer from limited scale and diversity and are susceptible to data contamination. 2. The methodology is innovative; it creatively uses existing benchmarks and large language models (LLMs) to generate new, more diverse problems from those already in the benchmarks. 3. Figure 2 offers key insights by comparing LLM performance on HumanEval+ and EvoEval, revealing that LLMs fine-tuned for instruction-following may perform well on HumanEval+ due to data leakage, yet underperform on EvoEval. 4. The evaluation is thorough, assessing 51 LLMs across the new benchmark, providing a robust experimental section. The analysis of the results is detailed and insightful. 5. The appendix of the paper is detailed, providing comprehensive supplementary material that supports the findings and methodology.

Reasons to reject

1. The EvoEval benchmark leverages GPT-4 for problem creation, which is also used in instruct-finetuning datasets for some evaluated LLMs. This could lead to biased performance estimates for these models, and may also skew the results that some key findings in this paper is based on. It would be beneficial for the paper to clearly indicate which models have even been trained on GPT-4-generated data, perhaps using specific symbols or annotations. 2. Typo: "DeepSeeker" -> "DeepSeek-Coder"

Questions to authors

N/A

Summary

This paper introduces a program synthesis benchmark called EVOEVAL for a comprehensive evaluation of LLM coding abilities. It aims to address the reliability limitation of previous benchmarks due to their potential data leakage issues. The proposed benchmark is created by evolving existing benchmarks into different targeted domains. In evaluation, it comprehensively considers 51 different LLMs across all benchmarks in EVOEVAL and the experiment results provide insightful and interesting findings.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

* This paper effectively tackles the shortcomings of prior code generation benchmarks by introducing a more comprehensive and robust benchmark to mitigate the potential data leakage problem. By doing so, it significantly improves the evaluation of LLMs' code generation capabilities across diverse aspects. I would anticipate this benchmark will be widely adopted by the community. * The experiments conducted are thorough and offer insightful findings. * This paper is very well-presented.

Reasons to reject

N/a

Questions to authors

N/a

Summary

The current research piece introduces EvoEval, a new benchmark constructed by “evolving” (i.e. modifying) existing problems from HumanEval on different aspects, making them more difficult, more creative in their description, changing the problems in a subtle way, combining different problems into one or make the models to have (or produce) a helper function to solve the problem. In addition, two extra evolutions were presented, rephrasing problem descriptions, while preserving their semantics, by evolving in a more concise or in a more verbose way. Problems were created using HumanEval as a starting point and prompting GPT-4 for their modification. The resulting benchmark contains 828 problems across 7 different benchmarks. The benchmark was evaluated on 51 open and proprietary LLMs with coding capabilities. Finding results interesting and new. While all the LLMs drop in performance, some of the best performing models on HumanEval drop significantly on EvoEval. Interestingly, some models have a significant drop in performance on subtle changes on their description. The overall assessment of this work is good. This new benchmark provides new insights to the scientific community in regards to code synthesis while introducing a new way to assess the quality of such models.

Rating

8

Confidence

4

Ethics flag

1

Reasons to accept

Strong work. Clear, and easy to read. A new benchmark bringing new insights to the scientific community. Proper and extensive evaluation, always being compared with previous benchmarks, HumanEvan in this case.

Reasons to reject

No reasons to reject

Questions to authors

Icons from Figure 5 are pixelated

Summary

This paper presents EvoEval, an evaluation benchmark for code generation tasks. The method converts existing problems into different tasks, with 7 different types of modifications. The EvoEval benchmarks are challenge and the performance of many LLMs has dropped using the benchmarks. The paper further points out some interesting trends, such as the instruction-following LLMs are very sensitive even to subtle changes. Overall, this paper identifies and attempts to solve a valuable problem: the contamination and potential overfit of LLMs on existing benchmarks. Theoretically, the method can be further used to create new benchmarks in the future. *Writing* Due to the relative complex setup and multiple types of “evol” methods, some explanations of the methods are a bit hard to follow. For example, the paper explains “COMBINE-NAÏVE” clearly, yet how “COMBINE”is done is a bit unclear.

Rating

8

Confidence

4

Ethics flag

1

Reasons to accept

- The benchmark presented by the paper is valuable to the community. Evaluation is one of the key components for advancement in AI. - The paper presents interesting finding of the current landscape of the models, by performing experiments 51 different variations of LLMs. These trends, such as the ranking change, variations related to subtle changes, point out some potential weakness of the language models. - The experiment is comprehensive and provide valuable insights

Reasons to reject

No particular reasons to reject.

Questions to authors

As mentioned in the text, models such as WizardCoder performs more robustly since it is trained on Evolved data. Would you consider models that are trained on similar evolution methods as a different form of overfitting?