NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness

Summary

In this paper, the authors introduced a new benchmark called NoFunEval, focusing on code generations against other non-functional qualities of codes such as security, efficiency, etc. In addition, the authors also provide a new prompting technique called Coding Concepts (CoCo) to prompt the models with the specification of the non-functional requirements.

Rating

5

Confidence

4

Ethics flag

1

Reasons to accept

- The benchmark is a timely direction to address the arising research of code generation and the need to evaluate diverse qualities of codes beyond functional correctness. I found the proposed list of non-functional requirements quite comprehensive, including the code latency, resource utilization, runtime efficiency, maintainability, and security. -Quite comprehensive evaluation with many experimental results from strong base code LLMs, including StarCoder, WizardCoder, Mistral, CodeLLama, etc. The results are fully described and analyzed with interesting insights e.g. generation vs. classification performance.

Reasons to reject

- The evaluation metrics are not ideal in some cases. For instance, in latency and resource utilization, the performance is evaluated by only DiffBLEU against the ground-truth code. It would be better to have a metric that actually measures the impacts of the generated code e.g. by real runtime, or utilized resources, even in a sandbox/ independent execution environment. - The overall number of samples in each task is quite small, ranging from 47 to 145 samples only. Therefore, it would be quite hard to generalize the results to more general languages or coding scenarios. - The prompting technique CoCo is rather trivial. Using the list of detailed hints/ guidelines, the models are expected to improve in their generation outputs. However, I am uncertain how these guidelines were obtained. If it was not explained in the paper, please describe how these hints/ guidelines were obtained. Are they extracted from data sources or manually written by humans?

Summary

The paper introduces a new benchmark, NoFunEval, to evaluate language models of code (code LMs) based on their ability to handle non-functional requirements such as efficiency, security, and maintainability, instead of simply focusing on functional correctness. The paper proposes a prompting method called Coding Concepts (CoCo) to help communication between the developers and code LMs. Twenty-two code LMs are tested using this new benchmark revealing significant shortcomings in their performance and indicating fundamental flaws in their training setups.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

The non-functional requirements are important to code LM in real applications. The experiments are extensive to show the potential direction.

Reasons to reject

The CoCo prompting needs expert-provided hints which is expensive to acquire in real scenarios.

Questions to authors

Given various problems in the dataset, how would you efficiently generate the "concepts" for each problem? Are the concepts different from problem to problem?

Summary

The authors propose and construct an evaluation suite for code language models to test their performance with respect to non-functional requirements. The authors claim to release the required code and data to the public, providing a valuable benchmark that, as the authors show, at least existing models do not perform well on.

Rating

8

Confidence

3

Ethics flag

1

Reasons to accept

- Authors formalize an important task (non-functional requirement related performance) and construct a benchmark dataset for it - Authors release code and dataset - Authors show that existing models do not perform well on the task

Reasons to reject

Authors restrict evaluation to a small number of model providers (both closed and open weights). Given authors finding that GPT-4 does not perform well on the proposed benchmark, comparison to other closed-weight models such as Claude or Gemini would have been interesting.

Questions to authors

- Please include samples from every task and prompt type in the appendix

Summary

This paper discusses the limitations of current benchmarks for evaluating code LLMs, which primarily focus on functional correctness and use test cases to evaluate the correctness of the generated codes. The authors propose a new benchmark called NoFunEval to conduct more code evaluations. They evaluate twenty-two code LLMs and find that they struggle on the new benchmark, indicating deficiencies in their training. Overall, this paper is well-written and clear.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

1. This paper builds a new dataset to better evaluate the coding ability of LLMs. 2. This paper conducts lots of experiments on different LLMs. 3. This paper is well-motivated, well-written and clear.

Reasons to reject

1. My main concern is that there are some code benchmarks, that also aim to evaluate the coding ability of LLMs at the Non-functional level, such as SWE-bench[1], CoderReviewer[2], and RepoCoder[3]. In this case, this paper should discuss the differences between the NoFunEval and existing benchmarks, such as the new evaluation metrics, evaluation on different dimensions, and code optimization. 2. The link can not be opened. 3. The statistics of the dataset on different program languages should be shown. [1] Can Language Models Resolve Real-world Github Issues? [2] CodeReviewer: Pre-Training for Automating Code Review Activities. [3] RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation.

Response to Authors

Even though the authors have claimed that the work is different with other benchmarks. But, I think these benchmarks also show the contribution to the No Function Eval. This paper should carefully revise to avoid the confusing points.

Thank you for acknowledging our response. We would like to add a few clarifications.

Thank you for acknowledging our response. We would like to clarify that: - In Appendix A5, we report a high correlation between DiffBLEU and Execution accuracy for HumanEvalFix dataset, in addition to correlation with CodeQL scores for Security and Maintainability. While prior work has shown BLEU to correlate poorly with functional correctness in HumanEval and MBPP benchmarks, it's important to note that unlike BLEU, DiffBLEU is specifically designed for code-edits, and that it correlates strongly with Execution accuracy for HumanEvalFix (Pearson coefficient 0.978). We will move these details from the appendix to the main paper for more clarity in the revised version. - Providing additional hints via prompts like CoCo can be useful in settings where users are already familiar with the codebase they are editing. Our main objective in evaluating CoCo prompts is to understand how well different LMs can respond to additional hints.