Summary
This work proposes to instruction-tune LLMs for code by using commit messages that clearly describe the change to the code in an imperative style, along with the original and new programs. It also introduces an extended version of HumanEval that spans 6 languages and three tasks as a benchmark. The results indicate that the proposed commit-based dataset in combination with OASST (from prior work) forms the strongest fine-tuning mixture, yielding strong results when fine-tuning StarCoder and CodeGeeX.
Strengths
This work makes multiple substantial and useful contributions to the domain of code-specific instruction-tuning. As it notes, many successful recent approaches have been based on data from proprietary models. Using commit messages from permissive repositories offers a large dataset of relatively easy-to-acquire, and apparently moderately useful, data. The benchmark it contributes is also quite useful, moving substantially beyond the standard HumanEval set.
The work is quite well written and the supplementary material offers a very thorough view of the methods and results. Overall, this is a good paper.
Weaknesses
The methodology makes a number of decisions that are not particularly well explained or defended. While none of these are critical, several of these would benefit from additional ablations and analyses, or an acknowledgement as a limitation. In no particular order:
The discussion around CommitPack leads with the rather enormous size of the initial dataset (~4TB). While this is accurate, the version used in this work is just 0.05% this size (2GB). This work offers no validation or experiments involving the original set. This both creates the impression that the 4TB set is likely to very noisy, and suggests that this work should really primarily emphasize the 2GB portion as far as concrete contributions of this paper are concerned. That would involve amending text like in the contributions on P2 and in the abstract, which don't mention the size of the subset that was actually used at all. This also relates to the "orders of magnitude more" comment on P9, which seems quite inconsequential.
Continuing the discussion of instruction data, it is not clear from the work why just 5K samples were chosen for fine-tuning. Was the concern that the model would drift too far if more samples were used? It is also not clear why (a) StarCoder was granted 3 extra samples, but that seems pretty inconsequential, and (b) OASST used nearly twice as many examples. Is that because the OASST samples are of higher quality? Was an ablation performed to choose these ratios? This is perhaps a particularly salient issue because one takeaway from this work is that CommitPackFT is not usable on its own. The experiments only ever report results of CommitPackFT + OASST (or similar combinations). This combination mostly boosts "Fix" performance; the results on the other two datasets are about even or worse than just using OASST by itself (Tab. 13). Perhaps the idea is that OASST is of very high quality but too small (are the ~8.5K samples used here the entire usable available subset?), so adding reasonably good samples increases performance? In any case, as is, the impression I get is that CommitPackFT isn't of particularly high-quality, but helps because it provides a bit more coding knowledge to OASST. More experiments could help contradict (or proof) that impression.
The work makes a rather strong argument, that building on HumanEval is a positive since it is so common that it is typically filtered from training data. That argument seems quite challenging to back up, given the absence of insight into how many LLMs are trained and anecdotal observations that pretrained models perform worse on new coding problems (e.g. from CodeContests). I would suggest a more moderated discussion of this choice, acknowledging the use of HumanEval as a potential limitation.
StarCoder is highlighted as performing very poorly in the explanation task, because it is unable to generate explanatory text. I wonder if this overlooks the obvious: query StarCoder to predict a docstring/javadoc/other comment using its FiM capabilities. The model has naturally not encountered samples were code is followed by a request for an explanation, but it would likely do relatively well when prompted to predict a comment above or at the start of a method. Although this somewhat stretches the definition of an "explanation", it does give the models more of a chance than the current setup.
Appendix C notes that inputs were filtered to just 768 tokens in length, for the complete before-code + message + after-code series. That is a very low limit by modern language modeling standards. Tab. 8 reinforces that this leads to very short programs compared to the natural distribution of commits. What was the motivation for these limits, and how does this impact the dataset's usefulness for downstream use-cases, where programs are rarely as short as those in the HumanEval suite?
Minor notes:
- The center panel in Fig. 3 might benefit from a visual improvement. Without reading the text, it was not clear to me what was happening here; as presented, it looks like the model is prompted with the code, predicts text, than is prompted with more text and predicts the code again, all in one input sample. Perhaps consider inserting a blank space between the two tasks (i.e., before the second model input) and moving the arrow to connect between the two halves.
- Sec. 3, P5: what experiments were conducted to confirm that GPT-4's accuracy only varies by 2% when sampling 1 sample vs. 20? I would imagine that the variation could be quite a bit more on some tasks. It might be worth sampling, say, 2-5 samples to balance cost and precision.
- It would probably help the work to also report BLEU/METEOR scores on the explanation generation task. This might offer a complementary view to the proposed automated metric, which is potentially somewhat lossy.
- A few odd notes from the appendix: why was "can’t you see i’m updating the time?" used as a prominent commit message filter (Tab. 5)? This doesn't seem to be a common phrase. Why was only data up till 2016 used? There are archives with more recent GitHub data.
Questions
To avoid redundancy, please consider the questions raised in the weaknesses section above. Primarily, focus on the question of CommitPackFT's value given the relatively limited results (e.g. compared to OASST) and the various places where limitations may need to be acknowledged more clearly.