Summary
This work propose a method to fine-tune an LLM for code generation using RL. The feedback consists of two components: localized compiler errors to encourage the LLM to generate code which compiles, and a CodeBERT-based discriminator which tries to distinguish between the code generated by the LLM vs. the ground-truth solution (conditioned on the prompt). Results show improvements on several baseline code generation models (up to 1.5B parameters) for Java code generation.
Strengths
The idea to fine-tune an LLM for code generation using only feedback from static analysis only is, as far as I can tell, novel. The writing is also clear and well motivated. As code generation is a major application of LLMs, the work has potential for significant impact as well. The experimental results are also relatively comprehensive, including a slate of baselines for comparison.
Weaknesses
Prior works (see below) have used a combination of static and dynamic analysis as an RL reward for fine-tuning LLMs for code generation, which limits the novelty.
The decision to focus on Java, motivated by the authors for its static typing and availability of static analyzers, makes comparisons with existing works difficult, which have almost universally adopted Python as the language of choice. The benchmark datasets (MathQA and MBJP) were originally written in Python, and then transpiled automatically into Java. Even the dataset used in this work for finetuning, CodeNetJava, has a Python equivalent. Additionally, this work leverages feedback from the static analyzer consisting solely of the location of the compiler error, which should be available for Python as well.
For instance, RLTF [1], which is another RL for code generation technique, achieves 30.4 pass@1 on MBPP (the original python dataset) using the 770M CodeT5 as the base model. This compares with 6.6% for RLCF (this work) on MBJP (using the same base model, which starts around the same pass@1 of ~4% for MBPP).
Ideally, I would have preferred the experiments to have been done in Python, but at the very least, the related work should include a discussion of works which apply RL for code generation. For instance CodeRL [2] is mentioned as a baseline, but there is no comparison in the related work. As far as I can tell, the specific design (using a discriminator as well as returning the location of a compile error) is novel, but there are certainly parallels to prior work (for instance, CodeRL uses a critic to return localized information). Another highly relevant work is [3], which combines RL with an AST-based syntactic match metric as well as a dataflow graph based semantic match metric (both of which are static rather than dynamic analyses).
Finally, I'm not sure the ablation study for CodeRL is a fair comparison, as CodeRL includes a number of other components beyond simply an RL reward for compile errors (such as the aforementioned critic network). This can be addressed by renaming the ablation to something other than CodeRL.
[1] Liu, Jiate, et al. "RLTF: Reinforcement Learning from Unit Test Feedback." arXiv preprint arXiv:2307.04349 (2023).
[2] Le, Hung, et al. "Coderl: Mastering code generation through pretrained models and deep reinforcement learning." Advances in Neural Information Processing Systems 35 (2022): 21314-21328.
[3] Shojaee, Parshin, et al. "Execution-based code generation using deep reinforcement learning." arXiv preprint arXiv:2301.13816 (2023).
Questions
Can you provide a comparison of your methods with the 3 works cited above?
Can you ablate the localization aspect of the compiler feedback? i.e., just return -1 for compile errors (while maintaining the discriminator).
It would also be good to swap out the learned discriminator with the DFG-based metric from [3] above.