AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Data

Summary

This paper presents an improved pipeline for training LLMs for code generation. Their method incorporates prompt modifications using hindsight tuning to modify prompts to better align with the associated code. In addition, they introduce improved data filtering and additional training tasks which they find give moderate additional performance boosts. Models trained using their pipeline show better generation abilities than existing LLMs and code LLMs on existing benchmarks and retain better ability on natural language tasks

Strengths

The method is relatively simple (though somewhat costly) and appears to give performance boosts across tasks. Improving the performance of Code LLMs in ways beyond scaling up data is a valuable contribution to the community and this method has demonstrated utility. The authors compare to a variety of models and demonstrate the efficacy of their method on a variety of model sizes and families. Additionally, the authors give good ablation studies on what elements in their method contribute to this performance boost.

Weaknesses

The main weaknesses of the paper come in lack of clarity. If these points can be clarified, I would be inclined to raise my score: 1. The writing in this paper was not especially clear to me, particularly when it came to motivating introducing this method. When contrasting with prior work, it would help to give examples of the type of low code quality and diversity that has been observed. More concerningly, because these details were not clearly explained, it was not clear to me how this dataset improved upon prior work. 2. The selection criteria are unclear for what samples that were chosen to be improved using AlchemistPrompts. If it is only a random 5% of the data that needs to be improved, it is unclear to me why performance benefits do not continue for higher percentages. On the other hand, if the data is selected more specially for this, it should be made clearer.

Questions

1. Is the 5% of data modified by AlchemistPrompter chosen at random or targeted? If they were chosen randomly, was the experiment repeated? Finally, if they were chosen randomly, is there a chance that using a heuristic to choose them could improve performance further? 2. Why should we expect AlchemistCoder models to be better generalists in NL tasks when the diversity they have seen is mostly in the form of code data? Though there is more diversity in this dataset, it is still domain specific, which makes the catastrophic forgetting hypothesis seem unlikely to me. Do you perform any further experiments on this?

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors explain adequately.

Summary

The paper presents a series of Code Large Language Models (LLMs) named AlchemistCoder, which are fine-tuned on multi-source data to enhance code generation and generalization capabilities. The authors address the limitations of previous Code LLMs that were typically fine-tuned on single-source data, which lacked diversity and quality, by introducing AlchemistPrompts, data-specific prompts generated through hindsight relabeling to harmonize different data sources and improve instruction-response pairs. Additionally, they incorporate the data construction process into fine-tuning as code comprehension tasks, including instruction evolution, data filtering, and code review. Extensive experiments demonstrate that AlchemistCoder outperforms models of the same size and rivals or surpasses larger models, showcasing its efficacy in refining instruction-following capabilities and advancing code intelligence.

Strengths

1. The paper is clear and well-written. 2. The contribution is simple, clear, and easy to be adopted. (Use GPT4-generated prompts to harmonize the domain gap) 3. The evaluation justifies the effectiveness of the proposed method. (The ablation study explains the effectiveness of each step.) 4. The improvement is impressive.

Weaknesses

I'm not fully convinced that the improvement is from "multi-source" data. AlchemistPrompts possibly introduced high-quality data from GPT-4. Table 4 also confirmed that the most gain is from that 5% AlchemistPrompt. If this is the reason, the comparison with other models weaker than GPT4 is not completely fair. It also means the proposed method is weak in relying on a strong model to supervise.

Questions

For different portions of AlchemistPrompts other than 5%, does the general increasing trend of accuracy on multi-source data in Table 4 maintained?

Rating

6

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

See weaknesses.

Summary

This work improves upon past work developing code LLMs with a focus on intervention on the data used to instruction tune the models. Specifically, the key insight in this work is that past works have usually relied on single source data for fine-tuning, but this can come at a drawback of quality and diversity. To reduce these issues, the authors use multi-source data. This in turn leads to a challenge where the same question can elicit multiple responses, calling for the need for AlchemistPrompts that "harmonize" the sourced. A second source of gains for the AlchemistCoder models series comes from construction of code comprehension tasks. Various experiments are performed to demonstrate the method is very effective in achieving performance as good as models from a larger model size.

Strengths

1. Clear problem identification: I liked that the authors identified a problem in existing literature training code LLMs pertaining to single source code data. They performed subsequent steps to mitigate this gap with single-source data, then harmonizing it. 2. Strong results: The alchemistcoder model series punches much above models of the same parameter size, as evident from results in Table 1 and Figure 1. 3. Wide range of evaluations: The authors test their model capabilities on both code generation tasks, and standard benchmarks, and the performance gains stay across the board. 4. Analytical experiments on data composition: I liked the analysis in Figures 5, 6 and 8 giving more perspective to the reader on the differences between AlchemistCoder data and data from past work.

Weaknesses

1. Missing ablations: Many of the comparisons are nor iso-compute (different models trained for different time). I believe an important aspect of this work is to clearly show model performance w.r.t. various ablations performed: 1. Multi-source 2. +Harmonization. 3. +Code comprehension. 2. Python-based evals: Even though multi-source data is used, all the evaluations are limited to python based evaluation. I would have liked to see more analysis on how the multi-source data impacts model performance in other languages.

Questions

1. Can you point me to this study: "Our ablation study indicates that the optimal performance can be achieved by incorporating AlchemistPrompts into only 5% of all the samples, striking a balance between the diversity and domain gap resulting from the fusion of multi-source data." 2. Is code comprehension data augmentation a novel contribution of this work?

Rating

6

Confidence

3

Soundness

4

Presentation

3

Contribution

2

Limitations

The generation of AlchemistPrompts relies heavily on GPT-4

Summary

This paper introduces AlchemistCoder, a series of code language models fine-tuned on multi-source data. The authors propose using "AlchemistPrompts" to harmonize inherent conflicts in multi-source code corpora and incorporate code comprehension tasks into the training process. The resulting models show improved performance on various code generation benchmarks compared to baseline models of similar size.

Strengths

1. The paper addresses an important challenge in fine-tuning code language models by utilizing multi-source data, which could potentially lead to more robust and versatile models. 2. Empirical results demonstrate significant improvements over baseline models on several code generation benchmarks, including HumanEval, MBPP, and DS-1000. The authors provide detailed ablation studies and analyses to validate the effectiveness of their proposed methods, including the impact of AlchemistPrompts and code comprehension tasks.

Weaknesses

1. The core concept of AlchemistPrompts lacks substantial novelty. It appears to be a variation of existing instruction evolution techniques, such as the Evol-Instruct. 2. While the performance improvements are notable, the overall approach of using multi-source data and harmonizing it is not fundamentally new in the field of language model fine-tuning. 3. The paper compares AlchemistCoder models (6.7B/7B parameters) with larger models, including 15B, 33B, and 70B models. While the results show that AlchemistCoder outperforms or rivals these larger models on several benchmarks (as seen in Table 1 and Figure 1), the paper does not provide a comprehensive analysis of the factors contributing to this performance gain. It would be valuable to have a more in-depth discussion on how much of this improvement can be attributed to the proposed method versus other potential factors such as the efficiency of smaller models or the quality of the base model used. This additional analysis would strengthen the paper's claims about the effectiveness of the AlchemistCoder approach relative to simply scaling up model size.

Questions

1. Can you provide more insights into the relative contributions of data diversity and the harmonization process to the overall performance improvement? Are there scenarios where one factor might be more important than the other? 2. How does the efficiency and effectiveness of the AlchemistPrompt approach scale with increasing model size and dataset complexity? Are there any computational or performance bottlenecks that might limit its applicability to very large language models or extremely diverse multi-source datasets?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have addressed some limitations of their work, such as the potential for hallucination in response to obscure queries. However, there are several other important limitations that could be more thoroughly discussed: 1. Scalability: The paper focuses on models with 6.7B/7B parameters. There's limited discussion on how the AlchemistPrompt approach and code comprehension tasks would scale to much larger models or more diverse datasets. The computational costs and potential performance bottlenecks for scaling up are not addressed. 2. Data Bias: The process of selecting and harmonizing multi-source data could potentially introduce or amplify biases present in the original datasets. The paper doesn't thoroughly address how these biases are identified, mitigated, or might impact the model's outputs.

Gentle Reminder

Dear ACs and Reviewers, We wish to express our gratitude for your endeavors to participate in the review process. Our utmost efforts have been dedicated to resolving the concerns articulated by all reviewers. We sincerely hope that you have had a chance to review our response to your comments. We kindly request that you get back to us as soon as possible since the discussion period is ending soon. We appreciate your time and effort in reviewing our work, and we are happy to provide any further information if required. Thank you for your ongoing support during this review process! Sincerely, Authors of Paper #10234

Thanks for the response

Thanks for the effort. I am willing to raise my score to 5 because the rebuttal effectively clarifies the unique aspects of AlchemistPrompts, distinguishing them from existing instruction evolution techniques. The authors' explanation of the different goals, methods, and practical effects is convincing. While I still have some minor questions about data complexity impacts, the overall approach seems promising for improving Code LLM fine-tuning with multi-source data.