SelfCodeAlign: Self-Alignment for Code Generation

Summary

- The paper introduces SelfCodeAlign, a fully transparent and permissive self-alignment pipeline for code generation in LLMs without relying on extensive human annotations or distillation from larger models. SelfCodeAlign generates instruction-response pairs from seed snippets, evaluates responses with test cases, and fine-tunes models based on successful executions. The approach shows superior performance over state-of-the-art methods, including GPT-3.5-Turbo-based distillation, particularly in HumanEval+ benchmark. The pipeline demonstrates effectiveness across various model sizes, emphasizing the benefits of self-generated data over teacher models with smaller performance gaps. - Overall, I feel that SelfCodeAlign is a very easy workflow to follow and I see much potential for such pipelines that do not depend on distillation or human annotations. I recommend an accept.

Strengths

## Originality - The paper adequately cites related work, clearly identifying gaps such as the lack of transparency in existing methods, which is a key motivation for their work. ## Quality - The submission is technically sound with both quantitative and qualitative analysis. - The authors provide detailed experimental results, demonstrating significant performance improvements over baselines. - The inclusion of both large-scale and small-scale model evaluations further strengthens the quality of the research. - In terms of ethical considerations, they have considered all terms of use as well as the data in code snippets. ## Clarity - Well organized paper, except for appendix. ## Significance - The results are highly significant, as SelfCodeAlign achieves performance improvements, notably surpassing models that are an order of magnitude larger in size. This work addresses the challenge of instruction tuning without human annotations or distillation, offering a scalable and transparent solution that advances the state of the art in code generation.

Weaknesses

## Originality - Perhaps similar to this paper [Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models](https://arxiv.org/pdf/2312.06585) ? Even if it is different, I think that this should also be part of your baseline comparison as well. ## Quality - The qualitative examples provided in the appendix are excessively long, which may overwhelm the reader and obscure the main differences and contributions of the methodology. It would be beneficial to reduce the number of examples or to shorten them, focusing on highlighting the key differences and improvements over baseline methods. Additionally, the examples are presented in black and white with no descriptions or annotations, making it difficult to discern their significance. Providing clearer, annotated examples with concise explanations would enhance the readability and impact of this section. - I do not see any weaknesses discussed in this work, for example, in what scenario do you think does this methodology not work? Why is the score still not perfect? (or for eg, below 80% accuracy)

Questions

- What about experiments/benchmarking on models that uses the GPT4 family as part of distillation? - Why did you only limit it to 33B, what about 70b? - Line 121: What is the difficulty for? It is subjective, so how does the difficulty aid the model/aid in finetuning of models? - Line 228-233, Table 7: Any reason why the increasing trend is not consistent, for eg for StarCoder2-15B, the score decreased when tuned with DeepSeek-Coder-33B data? - Line 232, Table 7: Why did you not fill up the blank cells just like the last row? This would have ensured that your statement is true for all models, because you are just basing off CodeQwen1.5-7B model only.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are stated. However, I think the authors miss one important point – the reliance on the seed snippets. The performance of the model is based on what it was finetuned on, so if the seed snippets are not sufficiently diverse or representative of the target tasks, then it might have resulted in a large drop in accuracy.

Summary

The authors proposed SelfCodeAlign that finetunes the model based on the filtered data generated by the same model itself. The authors conduct experiments to show that SelfCodeAlign outperforms most open-sourced models that were finetuned on public code dataset.

Strengths

The code generation problem is important and the results (compared to models trained on public dataset) are promising.

Weaknesses

Compare to models that are distilled/trained on non-disclosed data, the performance of SelfCodeAlign is not as competitive. The presentation can be improved, see **questions**.

Questions

1. Can you properly highlight the row in table 1? 2. I would suggest to give a brief summarization of the component analysis after line 92. 3. It seems crucial to me to understand why SelfCodeAlign outperforms other dataset (for example, GPT-generated dataset), is there an analysis on how these datasets differ? Also, you mentioned distribution shift across different models in 4.1, is there a qualitative/quantitative comparison between the code generated by different models?

Rating

6

Confidence

2

Soundness

2

Presentation

2

Contribution

3

Limitations

Limitations have been discussed.

Summary

This paper introduces SelfCodeAlign, an entirely transparent and permissive pipeline designed for self-aligning code large language models without the need for human annotations or distillation. By applying SelfCodeAlign to CodeQwen1.5-7B, the authors generated a dataset containing 74k instruction-response pairs. They then fine-tuned CodeQwen1.5-7B using this dataset, resulting in SelfCodeAlign-CQ-7B, which demonstrates robust performance on the HumanEval+ benchmark.

Strengths

1. The performance is satisfactory: SelfCodeAlign-CQ-7B achieves a pass@1 score of 67.1 on HumanEval+, outperforming larger models like CodeLlama-70B-Instruct (65.2), which is a significant achievement. 2. The process is auto-mated: This paper introduces a novel self-alignment pipeline including concept extraction from seed code, task generation, multiple response generation, and execution validation. This approach is independent of human annotations or large model distillation, making it easy to be applied. 3. Scalability: Experiments demonstrate the method's applicability to models ranging from 3B to 33B parameters, showing good scalability across different model sizes.

Weaknesses

1. Lack of Diversity in Generated Tasks: While the method aims to produce a variety of coding tasks, it is unclear how this diversity is achieved or measured. There is a risk that the generated tasks may be biased towards certain types of coding problems, which could limit the model's ability to generalize effectively. 2. Overreliance on Self-Generated Tests: The method relies heavily on tests generated by the model itself to validate responses. This self-validation approach could result in a feedback loop where the model learns to create tests that are easy to pass, rather than generating truly challenging or comprehensive tests. The paper does not address how this potential issue is mitigated.

Questions

Refer to the weakness.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Refer to the weakness.

Summary

This paper proposes a pipeline for generating synthetic instruction tuning data. The method consists of the following steps: 1. data filtering is applied to seed coding data to select high quality examples; 2. base LLM is used to generate a set of coding concept and category based on the seed data; 3. base LLM is used to generate coding instruction, response and test; 4. generated examples are selected based on the code execution result.

Strengths

1. the paper focuses on using base model to generate synthetic data to self-improve, which is an interesting and useful angle for synthetic data generation 2. the method is evaluated on several different coding LLM benchmarks which shows the effectiveness of the method 3. there are also ablation experiments verifying the contribution of specific design choices in the framework.

Weaknesses

While using base model to self-improve is an interesting and useful direction, synthetic data generation could be improved by using a stronger LLM than the base model. It is not clear from the paper whether the proposed framework would be effective compared to previous methods if we use a stronger LLM to synthesize the data. The synthetic data generation could also be potentially improved by having multiple rounds of data generation process.

Questions

1. Have you tried this framework using stronger LLM to generate synthetic data? 2. Can you get even better performance by running several rounds of data generation with improved base model?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes