Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions

Summary

This paper proposes CanItEdit, a hand-curated evaluation benchmark consisting of 105 Python code editing problems, spanning three different categories: corrective edits, adaptive edits, and perfective edits. Each problem comes with two different types of natural language instructions: descriptive (entailing specific details for more explicit guidance) and lazy (minimal instructions that more closely resemble typical user queries). They further construct a training set for this task by filtering the CommitPackFT dataset and also mining additional examples from GitHub commits, where the instructions correspond to commit messages. They demonstrate out-of-the-box, open-source models (CodeLlama, Mixtral, DeepSeekCoder, StarCoder, OctoCoder) fall behind closed models (GPT-3.5-Turbo, GPT-4) by significant margins on this benchmark. By fine-tuning DeepSeekCoder-Base on their new training set, they observe improvements which appear to reduce the gap to a certain extent.

Rating

7

Confidence

5

Ethics flag

1

Reasons to accept

- The authors propose an evaluation set that captures diverse use cases (e.g., editing based on detailed specifications like the descriptive instruction versus editing based on brief user queries like lazy instruction) as well as many different categories of edits: corrective edits (35 examples), adaptive edits (35 examples), and perfective edits (35 examples). I believe this is a novel and very useful contribution to the community as it allows more fine-grained evaluation of LLMs with respect to code editing tasks. - While I would like to get some more clarity on the details behind this dataset creation procedure (see below), the benchmark is hand-curated, making it likely a very high-quality evaluation set, relative to the few existing code editing datasets. This makes it a very valuable resource for the research community. - The authors defined a new metric ExcessCode which I believe is very clever and useful. Benchmarks like SWEBench which rely on execution metrics only do not penalize models for making unnecessary changes to the code. I think this metric is also an interesting contribution to the community. - The experiments are very comprehensive, spanning many different models and dataset ablations. This has highlighted important results with respect to the effect of model size on pass@1 and ExcessCode, and the gap between open and closed source models on this task.

Reasons to reject

- Very minimal details about the dataset creation procedure are provided. (Please see questions below). This makes it very difficult to understand how the dataset was constructed and consequently assess its quality. - The dataset is Python-only and does not allow evaluating across different programming languages. Missing references: - Automating Code Review Activities by Large-Scale Pre-training (https://arxiv.org/abs/2203.09095) - CoditT5: Pretraining for Source Code and Natural Language Editing (https://arxiv.org/abs/2208.05446) - Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM (https://arxiv.org/abs/2403.19114) - CodeEditorBench: Evaluating Code Editing Capability of Large Language Models (https://arxiv.org/abs/2404.03543). (Note: This can be considered contemporary work and I have just included it for reference.)

Questions to authors

- Please provide more details about the dataset creation procedure. For instance, how were the annotators recruited? What were the exact instructions provided to them? It is not clear how the annotators come up with problems. Are they from scratch or is a starting point provided to them? If a starting point is provided, how is it derived? Do these annotators write both the descriptive and lazy instructions. Please explain in more detail the following claim: "Upon completion of a problem, the lead generated sample completions to ensure that the failures and successes were reasonable and consistent with the problem's intent." - A common use case is updating documentation (e.g., comments, README); however, it is not clear how such documentation edits can be evaluated with test cases. Does this evaluation benchmark include such edits? - The following claim is made in the paper: "GPT-4, despite not being specifically trained on code-related tasks, surpasses…" Could you explain how you came to this conclusion? It does seem that GPT-4 was trained on a significant amount of code, including come commits data. - It's not clear why Table 4 was not presented like Table 3, with results for "Descriptive" and "Lazy" shown separately. Do the ablations perform differently for these different use cases? - Despite motivating the different categories of edits, the main paper does not dive into this at all. I see that some additional results are provided in Table 6 of the Appendix, but this does not include all models in Table 3. Could you share details on whether the same trends are there across these categories for all the models presented in Table 3?

Summary

This paper proposes a new benchmark to evaluate the instruction following ability of Large Language Models (LLMs) in code editing tasks. The evaluation exposes a significant gap between the capabilities of state-of-the-art open and closed models. For example, even GPT-3.5-Turbo is better than the best open-sourced model at code editing tasks. This paper also collects a new dataset named CANITEDIT comprising 105 meticulously constructed Python code editing challenges.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

1. This paper proposes a new benchmark containing 105 Python code editing challenges to evaluate the ability of LLMs in code editing tasks. The topic is interesting and the contributions are significant. 2. Several popular open-sourced code LLMs such as CodeLlama and DeepSeekCoder have been evaluated and compared with strong closed-sourced LLMs such as GPT-4. 3. The presentation is clear and the writing is easy to follow.

Reasons to reject

1. The evaluation metric is limited as there are only pass@k and ExcessCode in metrics. Authors might propose some refined assessment techniques for code editing tasks or add human evaluation. 2. Similar contributions with Guo et al.[1]. Authors might consider to compare with this work if possible. 3. The code languages types are limited since there is only Python language. [1] Guo et al., CodeEditorBench: Evaluating Code Editing Capability of Large Language Models.

Summary

The paper proposes a novel benchmark called CANITEDIT, which is designed to evaluate the instructional code editing skills of Code LLMs. It comprises 105 manual code editing tasks, each accompanied by two sets of natural language instructions and a comprehensive test suite for validation. Additionally, it presents a specially tailored training dataset for code editing. Through extensive experiments, the work highlights a significant performance contrast between closed and open models among state-of-the-art Code LLMs. Moreover, the presented training dataset has been shown to be effective in improving code editting capabilities of models at various sizes.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

* The proposed benchmark is novel. It expands the evaluation scope of code LLMs to include instructional code editing skills, an often overlooked aspect in their assessment. * The evaluation is comprehensive and provides insightful findings. This work underscores a significant performance contrast between closed and open code LLMs. Moreover, experimental results verify the efficacy of the curated training dataset in enhancing code editing capabilities across diverse model sizes. * This paper is well organized and presented.

Reasons to reject

* A potential concern arises from the limited scope of the benchmark, which encompasses only 105 manually crafted instructional code editing problems. This constraint could result in a restricted coverage of domains within the evaluation.

Questions to authors

See the comments above.

Summary

This paper proposes a benchmark to evaluate the instruction-following code editing capability of selected baselines. - The authors include several kinds of editing instructions in the design. - The authors provide an instruction-tuning training dataset bundle specifically for code editing enhancement.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

Compare to the portion of editing tasks in code-related conversations (19%) , the research on such specific kind of task is absent for LLMs. We need to study more about the LLMs on the capability of manipulating existing codes, distinguished from direct code generation from natural languages. - The authors provide a compact hand-crafted code editing instruction dataset for benchmarking. The curated commit-based fine-tuning dataset would be a useful resource for future research. - The proposed benchmark adapts the existing *corrective*, *perfective*, and *adaptive* editing task classification system, and further uses a *descriptive-lazy* instruction categorisation dimension, which is a concise fit for LLM research. Based on which, there are in-depth analyses of the results appended.

Reasons to reject

- **Quantified correlation with the code generation capability.** Although this paper focuses on the code editing task, the authors should also include explore the correlation between the general code generation and editing performances of the baselines in the main text, given that there is assumption that the former could be the basis of the latter. - **Lacking of human evaluation.** It would be better if the authors could provide the performance of human annotators in the benchmark as a guideline for the (half) open-ended problems.

Questions to authors

"These suites were designed to rigorously evaluate whether the ’after’ segment met the problem requirements while ensuring the ’before’ code did not." -- if all the "before" segments do not met the requirements, do they still fall into any of the three task categories? Can the authors provide clarification on the "relatively small" and "quite low" from "The mean Levenshtein distance between the ‘before’ and ‘after’ code segments is 197.1 characters, indicating that the changes are relatively small. We also analyze the distribution of the commit message lengths, and find that the mean token count is 10.1, which is quite low."? Is there any overlap between the two training sets?

Response to rebuttal

Thanks for your detailed response. I will be keeping my score the same.