CodeRosetta: Pushing the Boundaries of Unsupervised Code Translation for Parallel Programming

Recent advancements in Large Language Models (LLMs) have renewed interest in automatic programming language translation. Encoder-decoder transformer models, in particular, have shown promise in translating between different programming languages. However, translating between a language and its high-performance computing (HPC) extensions remains underexplored due to challenges such as complex parallel semantics. In this paper, we introduce CodeRosetta, an encoder-decoder transformer model designed specifically for translating between programming languages and their HPC extensions. CodeRosetta is evaluated on C++ to CUDA and Fortran to C++ translation tasks. It uses a customized learning framework with tailored pretraining and training objectives to effectively capture both code semantics and parallel structural nuances, enabling bidirectional translation. Our results show that CodeRosetta outperforms state-of-the-art baselines in C++ to CUDA translation by 2.9 BLEU and 1.72 CodeBLEU points while improving compilation accuracy by 6.05%. Compared to general closed-source LLMs, our method improves C++ to CUDA translation by 22.08 BLEU and 14.39 CodeBLEU, with 2.75% higher compilation accuracy. Finally, CodeRosetta exhibits proficiency in Fortran to parallel C++ translation, marking it, to our knowledge, as the first encoder-decoder model for this complex task, improving CodeBLEU by at least 4.63 points compared to closed-source and open-code LLMs.

Paper

Similar papers

Peer review

Reviewer BqwA6/10 · confidence 4/52024-07-07

Summary

This work intorduces an encoder-decoder transformer model, CodeRosetta, translating between programming languages and also their high-performance computing (HPC) extensions parallelly. The authors claim that CodeRosetta outperforms baselines and general LLMs on C++ to CUDA and Fortran translation tasks.

Strengths

1. In the translation task from C++ to CUDA and Fortran, the paper claims that the CodeRosetta 0.8B model outperforms much larger language models, including GPT-4, Gemini-Ultra, and StarCoder 15B. 2. This research employed various techniques, such as Abstract Syntax Tree (AST) Entity Recognition, Noise Injection, Token Dropping, and Language-specific Token Insertion, combined with supervised fine-tuning to enhance the model's performance on the code translation task.

Weaknesses

1. Both BLEU and CodeBLEU might not serve as accurate translation performance indicators [1, 2]. These semantic metrics struggle to capture the nuanced behaviors of code. While Compilation Accuracy appears to be a robust metric, it lacks a formal definition in the paper. 2. The functional consistency of the original code and the translated code is more important than whether the translated code is compilable. Adding functional correctness evaluation may make the experiment sound more reliable. 3. For the C++ to CUDA translation task, the paper notes that the model is trained on an unpaired collection of C++ and CUDA source files. This training method can explain how the model learns the knowledge of each individual language. However, a more detailed explanation is needed to clarify the source of the model's language translation capability. [1] Evtikhiev, M., Bogomolov, E., Sokolov, Y., & Bryksin, T. (2023). Out of the bleu: how should we assess quality of the code generation models?. *Journal of Systems and Software*, *203*, 111741. [2] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., ... & Zaremba, W. (2021). Evaluating large language models trained on code. *arXiv preprint arXiv:2107.03374*.

Questions

I am unfamiliar with the programming language compilers, but AST conversion should vary for C, CUDA, and Fortran. CodeRosetta may be unable to distinguish different languages and much less to generate executable code? Additionally, finetuning with synthetic data from language models can be considered as supervised learning, which may conflict with the paper topic?

Rating

6

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors addressed limitations in their work.

Authorsrebuttal2024-08-11

Thank you! --- Part 1

Dear Reviewer BqwA, Thank you for your quick response, your thoughtful feedback, and the opportunity to address your questions. --- ## Metrics and Compilation Definition **ChrF Metric:** Yes, ChrF refers to the Character n-gram F-score. We apologize for not providing the full name earlier. We used the HuggingFace Evaluate Metric library with the default $\beta$ value set to 2. We used the ChrF metric following the recommendations from the *“Out of the BLEU: How Should We Assess Quality of the Code Generation Models?”* paper, which states that: *``ChrF and ROUGE-L are the best-performing metrics for the assessment of code generation models among the metrics we consider.``* We have also included the results using the ROUGE-L metric: | Model | CodeRosetta | GPT4 | Gemini Ultra | Gemini Pro | |---------------|-------------|-------|--------------|------------| | Rouge-L(CUDA) | 82.12 | 63.37 | 69.27 | 69.82 | **Compilation Accuracy:** We acknowledge the limitation of using compilation accuracy as a metric in code translation, as it is possible for a model to generate code that compiles successfully but diverges from the intended functionality. To partially address this, we conducted a manual evaluation of 30 generated code samples to compare their behavior against reference implementations. While we fully understand the limitations of such evaluations, we found that the functional correctness of generated code was preserved in the majority of samples (~93%). We also did run those generated codes against the reference and found them to be compilable and functionally correct producing the same outputs with the same inputs as the reference. In addition, the use of the compilation accuracy metric enabled us a direct comparison with BabelTower (ICML 2022), especially given that their codebase is not open sourced. --- Following this discussion, we will dedicate a section in our paper that covers the limitations of existing metrics in code translation and our efforts to partially address these limitations through manual inspection. We will also include the samples in the Appendix. Taking all of these additional results into account, we believe we have made our best effort to address this concern, though we acknowledge it may not be perfect. That said, we would be more than happy to include any additional metrics you suggest that may better characterize the performance of our work. --- ## Fine-Tuning `Paper-Tables-1 & 2` report the results from the fine-tuned model. `Paper-Table-3-Page-9` reports the results from the pre-trained model. **Discussion on Fine-tuning for Code Translation w/o Verification in Supervised Manner.** In code translation, paired data is scarce. However, our model benefits from a foundational understanding of code translation acquired through unsupervised and self-supervised pre-training (243K training examples). We show that finetuning, even on limited synthetic data—*without verifying the generated samples and their one-to-one mapping to input code in a supervised manner*—generated by larger models (merely 5K paired samples---less than 2% of total data points), can further boost the model’s performance. While synthetic data may introduce some errors (as large models can make mistakes in their translation), the combination of foundational understanding during pre-training and fine-tuning with such a small number of synthetic data points can lead to additional improvements.

Authorsrebuttal2024-08-11

Thank you! --- Part 2

## Model’s Translation Capability Thanks for this great question. As you mentioned, there are entities, libraries, keywords, and syntaxes from the source language (e.g., C++) that may not be valid in the target language (e.g., CUDA). `std::unique_ptr` and other C++ Standard Template Libraries (STL) belong to this category and must be avoided in the translation. The pre-training process in CodeRosetta (esp. Weighted Token Dropping) equips the model to gain a semantic understanding of source and target languages. It reduces the occurrences of invalid tokens in the translation. Nonetheless, there may be cases in which the model fails to correctly translate from common entities in the source code to valid entities in the target code. Looking into our dataset, there was no occurrence of `std::unique_ptr` in the test set. However, as the training set has `std::unique_ptr`, we intended to test the capability of CodeRosetta by developing a C++ code that uses `std::unique_ptr`, we made sure this sample does not exist in the training set: ### Reference C++ Code ``` void initializeArray(std::unique_ptr<int[]>& array, int size) { for (int i = 0; i < size; ++i) { array[i] = i * 10; } } ``` ### CodeRosetta CUDA Generation ``` __global__ void initializeArray(int *array, int size) { int i = blockIdx.x * blockDim.x + threadIdx.x; if (i < size) { array[i] = i * 10; } } ``` This sample is about array initialization. As can be seen, the model is able to provide the CUDA code. It removes the `std::unique_ptr` from the CUDA kernel as it is not supported in the device code. In CUDA, `std::unique_ptr` is called in the host kernel, which is the main function. Our model focuses on the generation of device function and hence is successful in this case. Moreover, from the test set, we checked where our model could potentially fail. Here is an example of a translation that CodeRosetta struggles with: ### Reference C++ Code: ``` void getRho(const int numOfNucl, const double* psi, const double* occNo, double* rho, const char debug) { *rho = 0; for (int i = 0; i < numOfNucl; ++i) { *rho += occNo[i] * psi[i] * psi[i]; } if (debug == 1) { printf("DEBUG ▁ print ▁ of ▁ RHO:\n▁ RHO ▁ = ▁ %f\nThis ▁ is ▁ the ▁ last ▁ line (RHO).\n\n", *rho); } } ``` ### CodeRosetta CUDA Generation: ``` __global__ void getRho(const int numOfNucl, const double *psi, const double *occNo, double *rho, const char debug) { *rho = 0; for (int i = blockIdx.x * blockDim.x + threadIdx.x; i < numOfNucl; i += blockDim.x * gridDim.x) { *rho += occNo[i] * psi[i] * psi[i]; } if (debug == 1) { printf("DEBUG print of RHO:\nRHO = %f\nThis is the last line (RHO).\n\n", *rho); } } ``` In this example, CodeRosetta, along with all larger models like GPT-4, Gemini Ultra, and Gemini Pro, failed to translate the code correctly. We verified the generated code by executing it. The line `*rho = 0;` initializes the variable rho to zero. However, if this kernel is executed by multiple threads across different blocks simultaneously (which is very likely), all threads will write to the same memory location. This results in a race condition where multiple threads are simultaneously updating the value of *rho without any synchronization mechanism, such as atomic operations or reduction techniques. As a result, each thread's write could overwrite others, leading to incorrect and unpredictable outcomes. To rectify this issue, `rho` should be set to 0 in the host code, and `atomicAdd` should be included in the computation. --- We hope these clarifications address your feedback and questions effectively and you find our rebuttal and additional discussion satisfactory. We would greatly appreciate it if these additional results and clarifications lead you to reevaluate our work. At the end, if there are any other lingering questions or requests, please do not hesitate to ask. Our goal is to work with experts like you to improve the quality of our paper and make our research more valuable for the community. Thank you, The Authors

Reviewer BqwA2024-08-11

Thank you for your effort

Dear authors, Thank you for your diligent effort and thorough response! The experiments and explanations you provided have satisfactorily addressed all of my concerns. As a result, I am pleased to elevate my score from 3 to 6 and advocate for the paper's acceptance. Good luck!

Authorsrebuttal2024-08-11

Thank you for your time, thoughtful engagement, and recognition of our efforts!

Dear Reviewer BqwA, Thank you for your thoughtful engagement and for taking time to carefully review our response. It means a lot to us. We appreciate your recognition of our efforts and are delighted that our explanations and additional experiments have addressed your concerns. We're especially grateful for your decision to increase the score and advocate for the acceptance of our paper. Your feedback has improved the quality of our work, and we believe this will benefit the broader research community. We are committed to revising the manuscript for the final version to carefully reflect our discussions. Thank you once again for your support. Best Regards, The Authors

Reviewer W1wi5/10 · confidence 4/52024-07-08

Summary

This paper introduces CodeRosetta, an encoder-decoder transformer model for unsupervised translation between programming languages and their high-performance computing (HPC) extensions. The main contributions include unsupervised code translation: CodeRosetta translates between programming languages (e.g., C++) and their parallel programming extensions (e.g., CUDA and Fortran). Customized pre-training and training objective: The model employs Abstract Syntax Tree (AST) Entity Recognition and customized Denoising Auto-Encoding (DAE) to learn parallel programming syntax and nuances. Experimental results demonstrate that CodeRosetta outperforms state-of-the-art methods in C++ to CUDA translation, with improvements of 2.9 BLEU and 1.72 CodeBLEU points, and a 6.05% increase in compilation accuracy.

Strengths

1. This paper introduces two new learning objectives: Abstract Syntax Tree (AST) Entity Recognition (AER) and customized Denoising Auto-Encoding (DAE) with weighted token dropping and insertion. 2. CodeRosetta can learn both the general syntactic structure of code and the specific nuances of parallel programming constructs without relying on language-specific metrics. 3. CodeRosetta outperforms the current state-of-the-art baseline models.

Weaknesses

- The evaluation only included BLEU and CodeBLEU, which are syntax-based metrics, and compilation correctness. It did not test the runtime correctness of the code, indicating a lack of completeness in the evaluation metrics. - In Section 5.2 Ablation Study, the ablation study did not include parts about Masked Language Modeling (MLM) and back translation.

Questions

- In Section 5.2 Ablation Study, Table 3: Ablation Study for C++ to CUDA, it was found that removing the fine-tuning with data from large models resulted in the most significant performance drop. Although the authors stated that this step is optional, the model's performance without this step did not surpass the baseline. - Section 3.4 Back translation for unsupervised refinement and Section 3.5 Finetuning with synthetic data from language models both mention generating new data using large models. There is a potential risk of data leakage during this process. It is crucial to ensure that the data generated by large models does not overlap with the test data, which could affect the validity of the test results. - Minor issues. Line 27, "C++ ↔ CUDA or C++ ↔ CUDA" should be "C++ ↔ CUDA or C++ ↔ Fortran"?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have addressed the limitations of their approach.

Reviewer djHa5/10 · confidence 4/52024-07-09

Summary

The paper presents CodeRosetta, a transformer model designed for unsupervised code translation and their high-performance computing extensions, such as C++ to CUDA and Fortran. By introducing novel pre-training objectives like AST Entity Recognition and customized Denoising Auto-Encoding, CodeRosetta effectively learns parallel programming syntax. The model demonstrates superior performance, surpassing state-of-the-art baselines metrics like BLEU / CodeBLEU and compilation accuracy.

Strengths

**Strengths** 1. CodeRosetta introduces novel pre-training objectives to solve translation tasks about CUDA and Fortran, such as AST Entity Recognition and customized Denoising Auto-Encoding. 2. CodeRosetta is one of the first to handle code translation of code HPC extensions, marking an advancement in code intelligence. 3. The model outperforms state-of-the-art baselines in key metrics, achieving higher BLEU and CodeBLEU scores

Weaknesses

**Weaknesses** 1. Although novel pre-training objectives have been introduced, the authors have not methodologically distinguished these from predecessors using AST or Denoising objects. Training with code structures is prevalent in code learning, as seen in models like CodeT5[1] and PLBART[2]. The authors should discuss these previous methods and variants of structural modeling, as referenced in papers like [3,4]. 2. The experimental section lacks some rigor. For instance, in the Educational value filtering, "randomly sampled 100,000 C++ files from Stack V2 and employed GPT-3.5 to assess their ‘educational value’ for learning C++ coding concepts" poses challenges to reproducibility. The use of GPT-3.5 to evaluate C++ coding concepts and the prompt engineering methodology warrant further clarification. Additionally, some experimental details in Table 4 are not clearly presented. 3. Upon reviewing the data composition of models like StarCoder (StarCoderDataV1), which includes 58,355 CUDA and 165,446 Fortran files, it appears that existing models like StarCoder or DeepSeekCoder could be directly employed for research. Significant engineering effort in the paper was focused on vocabulary issues, such as inserting CUDA-specific tokens, yet there is no discussion on how existing code LLMs handle out-of-vocabulary problems with languages like CUDA. [1] CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation [2] Unified Pre-training for Program Understanding and Generation [3] A Survey of Neural Code Intelligence: Paradigms, Advances and Beyond [4] A Survey on Pretrained Language Models for Neural Code Intelligence

Questions

1. Is there an error in the left part of Figure 4? 2. The phrase "compare CodeRosetta against their best fine-tuned model, StarCoder (15B)" is unclear. While StarCoder's pre-training data does include Fortran code, the notion of "best fine-tuned" is not well explained. Could the authors clarify this? 3. The performance of the three closed-source models in Table 4 is surprisingly low. Could the authors provide more details on the testing procedures?

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

I do not identify potential negative societal impacts in this paper.

Reviewer 9e247/10 · confidence 3/52024-07-13

Summary

The paper looks into the problem of unsupervised code translation with a particular focus on parallel programming languages like C++ to CUDA. It trains (relatively) small encoder-decoder models compared to current LLMs while achieving very strong performance on translation tasks. It incorporates program-specific terms in loss such as token frequencies, and AST structure to achieve this.

Strengths

1. Significance - This is an impactful problem and the resulting approach competes with very large LLMs with ~small models. The authors perform appropriate ablations and the proposed AST structure and token frequency-based denoising in loss are also effective. 2. Clarity - The paper is well-written and provides appropriate context and background around methods. 3. Originality - While people have tried including program-specific constructs in loss previously for encoder models applying it to the translation domain in novel. The denoising scheme based on language-specific token frequencies is intuitive and is proven effective from thorough ablations 4. Qualitative analysis of compilation errors - It was good to see authors performing an analysis of mistakes in model-translated samples. The distribution or errors and thus motivated future direction of using execution feedback does look promising.

Weaknesses

1. Evaluation and metrics. Perhaps a challenging aspect of the results is the evaluation benchmarks and datasets used. a. Benchmarks. The authors achieve considerably strong results on the C++ to CUDA evaluation sets with BLEU and CodeBLEU numbers higher than 75. Do authors perform some analysis on contamination or deduplication on the benchmark and collected training sets? b. Evaluation metric. Functional correctness has emerged to be the winning metric in the code generation space. However, it seems more challenging to apply that to the translation problems. High BLEU might not necessitate better translations. Similarly, comparability as a metric does not ensure that the translation is correct. While it might be hard to design functional correctness-based evaluation, the authors do not perform any other evaluation to assess the quality of translation beyond the, somewhat unreliable automated metrics. 2. Performance on proprietary LLMs. GPT-4 and Gemini-Pro depict considerably worse BLEU for some language pairs. It would be useful to highlight the failure modes since it might also highlight some issues in BLEU-like metrics. 3. Missing details on the datasets. The authors can provide more details on the datasets and benchmarks used in terms of document length and document counts. Similar statistics for the benchmarks and model generations would be useful to understand the complexity of the task.

Questions

1. The AST structure loss seems interesting. Do the authors believe it would also improve embeddings for monolingual encoder models? 2. Backtranslation-based unsupervised learning. The authors use back-translation to improve the translation performance of the model. However, for given languages pairs A and B (going from A -> B -> A), if the intermediate sample B is not constrained, the model can cheat and not do the translation job currently (as an extreme example, just copy the tokens from A). Can the authors clarify how the setup avoids such failure models? 3. What tokenizers do the authors use for these languages which are usually less well represented in the web data?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

4

Limitations

Authors have limitations in the the appendix.

Reviewer BqwA2024-08-10

Response to Author Rebuttal

Dear Authors, Thank you for your prompt response. **Metric Issue and Formal Definition of Compilation** * **ChrF Metric:** I assume 'ChrF' refers to the 'Character n-gram F-score'. Could you share the $\beta$ value used in your settings? Much like token-level metrics, the ChrF metric at the character level may also fail to accurately reflect the true code translation capabilities. * **Compilation Accuracy:** In natural language translation tasks, language models often generate specious responses. I am concerned that your model might generate code that compiles correctly but is functionally different from the intended solution. **Functional Correctness** * I understand that it is not feasible to conduct extensive experiments to evaluate the functional correctness of the generated code within the rebuttal period. I appreciate your efforts in manually analyzing the model's outcomes. **Fine-Tuning with Synthetic Data** * Could you clarify whether the scores reported in Tables 1 and 2 originate from the pre-trained model or the fine-tuned model? **Model’s Language Translation Capability** * Given that you are pre-training the model with unsupervised objectives, I recognize that the model can acquire monolingual capabilities, and perhaps some bilingual capabilities from frequently occurring common usages—the area where rule-based deterministic algorithms excel. However, I am curious about the model's ability to handle novel cases. For example, you mentioned that "some entities are not common among languages, such as pointers." How does the model handle the conversion of 'cuda::unique_ptr' to C++ code? **Differences in ASTs of Different Languages** * Thank you for providing an answer to my question.

Reviewer 9e242024-08-12

Thanks for the response

Thank you for performing the detailed analysis and not shying away from the limitations of BLEU. I would encourage the authors to add this discussion to the manuscript. Such incidents can add a fair amount of noise to the evaluations. However, the improvements achieved by the work are considerable and I remain cautiously optimistic for the paper.

Authorsrebuttal2024-08-12

Thank you for the support!

Dear Reviewer 9e24, Thank you for your thoughtful review and for recognizing the significance of our contribution. We appreciate your support and the time you took to engage with our detailed analysis. Your feedback has been instrumental in improving our work. We will incorporate the discussion on BLEU and the potential noise in evaluations into the final version of the manuscript, as you suggested. Thank you once again for your encouraging feedback and supporting our work. Best Regards, The Authors

Authorsrebuttal2024-08-13

Thank you for your time and recognition of our efforts!

Dear Reviewer djHa, Thank you for your thoughtful engagement with our work and for taking time to carefully read our rebuttal. We are glad to hear that our explanations and additional experiments have largely addressed your concerns, and we sincerely appreciate your decision to raise your evaluation. We apologize for the misunderstanding your point regarding Figure 4. Upon review, we identified a typo in the figure, and we have since corrected it to align with the text. We are committed to incorporating these additional results and discussions in the final version of the paper, ensuring that they accurately reflect our discussions. Thank you once again for your support and valuable feedback. Best regards, The Authors

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC