SemCoder: Training Code Language Models with Comprehensive Semantics Reasoning

Code Large Language Models (Code LLMs) have excelled at tasks like code completion but often miss deeper semantics such as execution effects and dynamic states. This paper aims to bridge the gap between Code LLMs' reliance on static text data and the need for semantic understanding for complex tasks like debugging and program repair. We introduce a novel strategy, monologue reasoning, to train Code LLMs to reason comprehensive semantics, encompassing high-level functional descriptions, local execution effects of individual statements, and overall input/output behavior, thereby linking static code text with dynamic execution states. We begin by collecting PyX, a clean Python corpus of fully executable code samples with functional descriptions and test cases. We propose training Code LLMs not only to write code but also to understand code semantics by reasoning about key properties, constraints, and execution behaviors using natural language, mimicking human verbal debugging, i.e., rubber-duck debugging. This approach led to the development of SemCoder, a Code LLM with only 6.7B parameters, which shows competitive performance with GPT-3.5-turbo on code generation and execution reasoning tasks. SemCoder achieves 79.3% on HumanEval (GPT-3.5-turbo: 76.8%), 63.6% on CRUXEval-I (GPT-3.5-turbo: 50.3%), and 63.9% on CRUXEval-O (GPT-3.5-turbo: 59.0%). We also study the effectiveness of SemCoder's monologue-style execution reasoning compared to concrete scratchpad reasoning, showing that our approach integrates semantics from multiple dimensions more smoothly. Finally, we demonstrate the potential of applying learned semantics to improve Code LLMs' debugging and self-refining capabilities. Our data, code, and models are available at: https://github.com/ARiSE-Lab/SemCoder.

Paper

Similar papers

Peer review

Reviewer H1xj6/10 · confidence 4/52024-06-17

Summary

This paper proposes SemCoder, a code LLM with code semantic-aware pre-training. The authors first use OSS-instruct to create a synthetic dataset PYX for model training, and a PYX-R dataset for training LLMs for debugging and repair. Then the authors train the SemCoder model with NL to code, forward monologue, and backward monologue tasks. Results on code generation datasets show that SemCoder outperforms a group of open-source LLMs and GPT-3.5 with fewer parameters. SemCoder also demonstrates better ability in execution reasoning.

Strengths

+The authors successfully trained a code LLM, and proposed novel approaches for both training data collection and training task design. +The forward and backward monologue is interesting and proven helpful.

Weaknesses

-In Section 2, I cannot find the examples for operational and abstract semantics in Figure 1 (“the middle yellow box” and “the extreme right box”). -The approach of collecting traces seemed limited to simple programs without external dependencies and types. I wonder if SemCoder can still benefit code generation/understanding tasks in real-world data.

Questions

- When building the PYX dataset, how did the authors choose the seed snippets? - Why do the authors choose DeepSeekCoder-6.7b as their backend model (instead of other code LLMs or its instruction-tuned version)? - Can the authors discuss why SemCoder-S significantly outperforms SemCoder on HumanEval, but achieves similar results on the rest tasks?

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The limitations are well-discussed. However, I'm still curious whether the proposed approach can be extended to real-world programs with more complicated dependencies and data types?

Reviewer MYsU7/10 · confidence 4/52024-06-21

Summary

Recent advancements in Code Large Language Models have primarily focused on code completion, yet these models often lack the ability to comprehend deeper semantics, such as the execution effects and dynamic states of code. This paper presents a pioneering approach aimed at enhancing Code LLMs' understanding of complex semantic tasks such as debugging and program repair. The methodology integrates high-level functional descriptions, local execution effects of individual statements, and the overall input/output behavior into the training of Code LLMs. This integration bridges the gap between static code text and dynamic execution states. The authors introduce PYX, a curated code corpus of fully executable samples with functional descriptions and execution tracing. They train Code LLMs to simulate human-like debugging by writing code and articulating execution behaviors in natural language. This approach culminates in the creation of SEMCODER, a specialized Code LLM with 6.7 billion parameters. SEMCODER demonstrates competitive performance against GPT-3.5-turbo in code generation and execution reasoning tasks, achieving 81.1% on HumanEval (compared to GPT-3.5-turbo's 76.8%) and 54.5% on CRUXEval-I (surpassing GPT-3.5-turbo's 50.3%). Furthermore, the authors explore the effectiveness of SEMCODER’s monologue-style execution reasoning versus concrete scratchpad reasoning, revealing a smoother integration of semantics across multiple dimensions. The study also underscores the potential of leveraging learned semantics to advance the debugging and self-refining capabilities of Code LLMs.

Strengths

+ Important area. + The authors integrate the code semantics into SEMCODER. + The performance is good.

Weaknesses

- Limited to Code Generation.

Questions

This paper introduces a novel strategy to train Code Large Language Models (Code LLMs) that significantly enhances their understanding of complex tasks such as debugging and program repair through the integration of high-level functional descriptions and execution effects. The authors have developed SEMCODER, a model that performs competitively with GPT-3.5-turbo in standard code generation. - Limitation to Code Generation: While SEMCODER excels at tasks related to code generation and execution reasoning, its application to other areas of software development, such as code comment generation, remains unexplored. I believe the code semantics may be helpful in these areas. Overall, the idea of this paper is good. I advocate to accept this paper.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

4

Limitations

Please see Questions, thanks.

Reviewer sURP6/10 · confidence 4/52024-07-11

Summary

This paper addresses the limitations of Code Large Language Models in understanding deeper semantics such as execution effects and dynamic states. It introduces a novel training strategy to enhance Code LLMs with comprehensive semantics, including high-level functional descriptions, local execution effects, and overall input/output behavior. Using a clean code corpus called PYX, the approach trains Code LLMs to write code and reason about execution behaviors using natural language, mimicking human verbal debugging. The results demonstrate the effectiveness of integrating semantics from multiple dimensions and improving debugging and self-refining capabilities.

Strengths

+ The idea of introducing different semantics to augment code LLMs is quite interesting. + Using various kinds of semantic information to enhance code LLMs is a commendable approach. + The experiments effectively demonstrate the model's capabilities across different aspects.

Weaknesses

- Some descriptions lack clarity. - There are no use cases demonstrating the model's usefulness. 1. In section 6.2, how did you fine-tune the base models? What are the data generated for NeXT and Scratchpad, including both forward and backward monologues? Line 331 only mentions the backward monologues. 2. In Table 2, what metrics are used for evaluating input prediction, especially how can you evaluate with your abstract input representations? Can you show some trace examples of backward prediction for different baselines in the table? It is not easy to understand why using abstract semantics works better. Some case studies would be helpful. 3. In Table 3, why was NeXT not selected as a baseline, given its focus on repair? What are the differences between CodeLlama with PYX-R and SEMCoder (-S)? Since the base models are different, why wasn’t SEMCODER fine-tuned on CodeLlama and Magicoder for fair comparisons?

Questions

Please check the questions in the weaknesses part.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N.A.

Reviewer GuSF6/10 · confidence 3/52024-07-12

Summary

The paper proposes PyX, a training dataset focusing on preserving the semantics of sample programs, and SemCoder, a coding language model trained with PyX with better ability to capture program semantics. The construction of PyX dataset comprises three phases to ensure the code samples involved are correct and self-contained. The dataset also contains pairs of buggy code, faulty traces and patches to improve the ability of LLMs in terms of debugging and self-refining. Based on the PyX dataset, a coding LM SemCoder is trained with a technique called Monologue Reasoning to improve the ability of code reasoning and comprehension in forward and backward directions. Comprehensive experiments are conducted to evaluate the performance of SemCoder, showing the SOTA performance and self-refining ability of SemCoder, along with the effectiveness of the training techniques involved.

Strengths

+ A novel and open-source dataset to tackle a widely concerned problem + A language model trained on the dataset with SOTA performance + Easy-to-understand and effective ideas and techniques for the training process + Clear presentation

Weaknesses

+ Hard to explain how monologue reasoning improves the performance of common coding tasks + Limited intuition on the experimental results, e.g. how does SemCoder conduct self-refining

Questions

+ During the construction of PyX, sample programs are filtered following certain criteria (e.g. Python built-in types only). However, user-defined types such as classes are crucial and frequently-used in real-world programs. Will this act impair the generalizability and applications of PyX and SemCoder? How can this gap be tackled if possible? + During the construction of PyX, a generator model is adopted to generate pairs of problems and solutions. Is there any potential bias that the model tends to generate outputs with certain limitations (e.g. low complexity, or high similarity to the training dataset of the model)? If so, to what extend does it eventually impair the generalizability of PyX? + It seems that the monologue reasoning technique (especially the forward one) is targeted at concrete inputs and outputs of programs. What about abstract inputs and outputs that conforms to certain specifications? Can the technique improve the summarization (of abstract inputs/outputs and specifications) ability of LLMs , and how? Such ability can be crucial under certain tasks such as documentation/specification generation. + How does monologue reasoning improve the code comprehension ability of LLMs? Does the effect of monologue reasoning rely on the explicit textual analysis in the outputs of LLM? More specifically, how does the improved code reasoning ability of SemCoder comes into play under common coding tasks such as code completion, especially when explicit textual analysis on input code is not allowed? I expect to see more analysis and intuitions on this regard. + Can the author provide more examples and intuitions on the debugging and self-refining process of SemCoder? What kind of information is given to the LLM as feedback? How does the LLM utilizes the information? To what extend can the LLM identify the root cause? How effective is the LLM in terms of fixing the bugs? + Also, can the debugging ability of SemCoder be adopted into tasks like program repair? Is it possible to conduct more thorough evaluations on this regard with benchmarks more targeted on APR?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

See weakness

Authorsrebuttal2024-08-06

## Debugging, Self-refining, and Program Repairing Here, we show an example of debugging: **Model Input** ``` Debug and Refine the Code: You are given a <Prompt> that describes a problem to be implemented in Python, <Fault & Trace> that the buggy implementation could not resolve the problem and fails the <Failed Test>, and the corresponding failed execution is traced and attached to code lines as comments. You should debug to identify the root cause of its failure. Finally, fix the bug and wrap the refined code in between [Refined] and [/Refined]. <Prompt> Implement a function that takes a list of potential energies, sorts them in ascending order, removes duplicates, and returns the indices of the unique sorted energies. Write a function `unique_sorted_indices(energies: List[float]) -> List[int]` that takes a list of potential energies as input and returns a list of indices corresponding to the unique sorted energies. <Fault & Trace> from typing import List def unique_sorted_indices(energies: List[float]) -> List[int]: energies_with_indices = list(enumerate(energies)) sorted_energies_with_indices = sorted(energies_with_indices, key=lambda x: x[1]) unique_sorted_energies_with_indices = list(set(sorted_energies_with_indices)) indices = [index for index, _ in unique_sorted_energies_with_indices] return indices # …Execution Trace… # … <Failed Test> assert unique_sorted_indices([10.5, 8.2, 10.5, 7.1, 8.2]) == [3, 1, 0] ``` **Model output** ``` The root cause of the failure in the provided code is that the implementation does not handle duplicate energies correctly. The code removes duplicates based on the entire tuple (index, energy) instead of just the energy value. This leads to incorrect indices being returned for unique energies. To refine the implementation: 1. After sorting the energies, iterate through the sorted list and keep track of unique energies in a separate list. 2. While iterating, also store the corresponding indices of unique energies in another list. 3. Return the list of indices corresponding to unique energies. def unique_sorted_indices(energies: List[float]) -> List[int]: energies_with_indices = list(enumerate(energies)) sorted_energies_with_indices = sorted(energies_with_indices, key=lambda x: x[1]) unique_energies = [] unique_indices = [] for index, energy in sorted_energies_with_indices: if energy not in unique_energies: unique_energies.append(energy) unique_indices.append(index) return unique_indices ``` Motivated by NeXT, we provide the problem description, faulty code and its execution trace, and the failed test to the model as input, and the model will first analyze the root cause of the bug, and then propose the fix plan, finally followed by the concrete fix. We recognize that evaluating the accuracy of debugging is tricky since it is described as a rationale in natural language explaining the localization, root cause analysis, and fixing plan altogether, which could not be automatically evaluated. Therefore, we choose to manually study a few cases. We observe that when SemCoder correctly fixes the program (e.g., the above case), the debugging rationale is mostly accurate in all components of its analysis. We will include more case studies and manual analysis in the revision of the paper to illustrate SemCoder’s effectiveness of debugging and program repair ## Relation to APR As shown in the above example the debugging and self-refine process includes bug localization, root cause analysis, refine planning, and the final fix, so it is quite similar to the setting of APR.. Considering the short context length of SemCoder, we found it challenging to directly evaluate SemCoder on repository-level APR benchmarks, such as Defects4J, which require lengthy contexts to analyze the bug.

Reviewer MYsU2024-08-12

Thanks. The paper is good. I keep my score at 7.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC