Response to Reviewer mVr1 (2/2)
> The paper lacks a comprehensive comparison with previous benchmarks about code completion.
Thank you for highlighting the need for a more thorough comparison with previous benchmarks. While our paper does include some discussion on 'code completion datasets' in related work section (Section 2), we acknowledge that this discussion might be somewhat limited, and we appreciate your guidance on enhancing this aspect of our work.
Firstly, as you pointed out, RepoBench distinguishes itself by focusing on repository-level tasks, which we believe is not only practical but also closely aligns with real-world applications. Also, we focus on using the most recent repositories to mitigate the issue of data leakage. In contrast to benchmarks that might focus on narrower datasets, RepoBench encompasses thousands of repositories. This extensive coverage is critical in evaluating modern models, especially considering that some may employ strategies like 'rephrasing' to achieve high scores on benchmarks like HumanEval. Our broad dataset scope ensures a more thorough and diverse evaluation.
Unlike previous retrieval tasks that often focus on simple semantic matching, our benchmark's code retrieval is contextualized within the code completion process. This adds a layer of practical significance, especially for projects aimed at enhancing code completion tools.
In light of your valuable suggestion, we plan to include a more detailed comparison of both retrieval and completion aspects in our revised paper.
> Why not using widely used metrics such as pass@k and CodeBLEU?
Firstly, our selection of EM and ES metrics simply aligns with prior research on code completion. For instance, [CodeXGLUE](https://arxiv.org/pdf/2102.04664.pdf), [LongCoder](https://arxiv.org/abs/2306.14893).
Regarding pass@k, we acknowledge its potential as an ideal evaluation metric due to its emphasis on executable code. However, our strategy prioritizes extensive data utilization to accurately reflect the capabilities of models across thousands of codebases. Implementing pass@k involves ensuring the executability of each piece of code, which is a complex task given the diverse environmental configurations and dependency issues common in library code. The logistical challenges associated with verifying the executability of such a vast array of code samples render pass@k less feasible for our study's scope. Nonetheless, we agree that pass@k could be a valuable addition in a more controlled or smaller-scale setting and will consider it for future research where applicable.
As for CodeBLEU, we appreciate your suggestion and have taken steps to incorporate it into our evaluation. Our latest experiments, as detailed in the Global Response, already include CodeBLEU as an additional metric. This inclusion should provide a more rounded evaluation of the models’ capabilities.
Despite these challenges, we believe that EM, ES and CodeBLEU are effective in reflecting the performance of models on dataset with a large scale.
> Expanding Model Selection for RepoBench-C Evaluation
Following your advice, we have conducted a test using the 7B version of SoTaNa. The results, which are detailed in the global response section, indicate that SoTaNa, despite being relevant, has limitations in code completion. We suspect these limitations are due to its foundation on the Llama model, which is not primarily a code model, and its fine-tuning on a context length of 512, not ideal for our settings of "long context" evaluation. Nonetheless, SoTaNa is definitely a relevant work and adds valuable insights to our study. Regarding the incorporation of more diverse Language Model Models (LLMs), our focus was primarily on base models. However, in light of your feedback, we have decided to broaden our scope by including CodeLlama in its various configurations (7B, 13B, 34B). Please check the added results in global response, where shows the impressive performance of CodeLlama.