Lemur: Harmonizing Natural Language and Code for Language Agents

Response to reviewer QCzC

Thank you for your reviews and comments to the paper! We are glad to hear that you appreciate the idea of harmonizing natural language and find our results solid and competitive. Below we address the concerns and questions raised in the review. --- **Weakness 1:** *Lack of Clear Presentation of Insights* **Response:** We appreciate the detailed feedback you provided on our writing and clarity. Based on these suggestions, we have highlighted the most important insights about harmony ability with specific descriptions in the introduction. Additionally, we have added error analysis as well as additional findings and insights. --- **Weakness 2:** Minor Typos in the Paper **Response:** We have corrected the typos in our revised version. Thanks a lot for pointing them out! --- **Question 1:** Reason for High Proportion of Python in Pre-training Data **Question 2:** Harmonization and the Text-to-Code Ratio Decision **Response:** We thank you for your valuable questions on our technical details! - The data mixture strategy is a complex trade-off between the model size, the vanilla performance of the model, data quality, learning efficiency, catastrophic forgetting etc. - To roughly estimate a good code-to-text ratio for composing the training corpora, we conducted a continue pre-training study with Llama on a set of different ratios and found that the ratio of 10:1 is an efficient ratio for the Llama model to transfer from text to text-code balance, which aligns with the findings of recent work like CodeLlama. - Among all programming languages, we focus on scripting or interpreted languages (Python, SQL, Bash, Perl, etc.) because agent models are often executed in interactive scenarios. Unlike compiled languages like C++, the interpreted nature of scripting languages allows immediate execution and easy modification, which is essential for dynamic interaction in language agent scenarios. - Due to computational limits, it is difficult for us to conduct comprehensive experiments and perform systematic studies on this scale. However, we hope that our open-sourced effective settings and training checkpoints can benefit the community for continual exploration. - We believe that a method to predict the optimal continue-pre-training data mixture ratio for a pair of domains to maximize their performance would be very meaningful and interesting, and it is still an open research question for the current large language model. We have added this explanation in the appendix of our revised version. Thanks for your valuable question!

Response to reviewer JHRb (1/2)

Thank you for your insightful reviews and comments. We are glad to hear that you found our evaluations comprehensive and considered Lemur as a valuable resource for developing advanced open-source agents. Below we address the concerns and questions raised in the review. --- **Weakness 1:** How the proposed training procedures balance these two abilities **Response:** - The strategy we use to balance the two abilities involves two-stage training. First, we train a good text model LLaMA-2-70B on code-intensive tasks, which aims to enhance its coding abilities. This gives Lemur-70B. Second, we finetune Lemur-70B to improve its instruction-following capabilities on both text and code scenarios. - From the results we present in Table 3 (which we also copy here for easier reference), it can be seen that Lemur-70B outperforms LLaMA-2-70B in code tasks, but maintains a good balance in text tasks. Additionally, thanks to its instruction-following capability, Lemur-70B-Chat achieves better performance in all scenarios. | Model | Text | | | Code | | | | | Average | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | | QA | Reason | Math | Python | | SQL | MCode | DS | | | | MMLU | BBH | GSM8K | HumanEval | MBPP | Spider | MultiPL-E | DS-1000 | | | LLaMA-2-70B | 68.9 | 51.2 | 56.8 | 30.5 | 45.4 | 60.0 | 24.4 | 11.3 | 43.6 | | Lemur-70B | 64.5 | 51.6 | 54.9 | 35.4 | 53.2 | 62.8 | 30.4 | 30.7 | 47.9 | | Lemur-70B-Chat | 65.3 | 61.9 | 66.3 | 61.0 | 55.5 | 62.5 | 42.9 | 34.5 | 55.0 | --- **Weakness 2:** Explanation or discussion of the performance differences with GPT-4 **Response:** We appreciate your attention to the differences between GPT-4 and our Lemur model! To better address your concern about performance gaps and guide the development of future language agent models, we added a 3-part human error analysis section in Appendix D in the revised paper. We hope this section can provide a detailed analysis of the three scenarios you mentioned, and also reflect the underlying reasons for the discrepancy between GPT-4 and our model. We summarize them as the following: - **Tool Usage ability:** We found that very complex and lengthy problems make it difficult for the models except GPT-4 to generate executable agent actions in the correct format. We found this problem when studying the M-MATH results. The long problems from MATH include solving equations described using LaTeX syntax, which is long and noisy. This misleads language models, which often generate unparsable responses or wrong actions. We hope to keep improving the ability of instruction following abilities in these scenarios. Here is part of our newly added Table 9: | Model | Execution error | Invalid action | |------------------------------|-----------------|----------------| | Lemur-70B-Chat | 29.00 | 26.00 | | gpt-3.5-turbo | 17.00 | 15.00 | | gpt-4 | 7.00 | 2.00 | - **Self-debugging ability:** Although the Lemur model is close to the performance of GPT-3.5-turbo in many environments of this scenario, we found that there is still a significant gap between Lemur and GPT-4 in challenging problems. We take InterCode-SQL as an example to study this issue. We categorize the problems of the SQL problems into different levels. According to the table, as the difficulty level increases, the performance gap between GPT-4 and Lemur gradually widens. This guides us to pay more attention to challenging cases. Here is part of our newly added Table 10: | Model | Easy | Medium | Hard | Extra | All | |------------------------------|-------|--------|-------|-------|-------| | Lemur-70B-Chat | 92.74 | 74.89 | 70.69 | 45.78 | 73.79 | | gpt-3.5-turbo | 92.74 | 74.89 | 67.24 | 43.37 | 72.82 | | gpt-4 | 95.16 | 82.96 | 86.21 | 68.67 | 84.14 | - **Exploring in Partially-observable environment:** We manually researched the CTF (Capture The Flag) tasks, which is a popular competition program originating from cybersecurity. We manually labeled the problems in 100 CTF tasks and divided each problem into 6 categories. We found that GPT-4 and Lemur perform significantly better in the "general skills" category than in domain-specific fields such as "cryptography" and "reverse engineering". This means current language agents methods easily fail in domain-specific scenarios, which guide the future research of language agents. Please refer to our newly added Figure 5. Please refer to Appendix D of our revised paper for more detailed information. We hope the added concrete detailed analysis can address your concerns!

Response to reviewer nTpx

Thanks for your time in reviewing and providing feedback for our work! We greatly appreciate your recognition of the effectiveness of the Lemur model and its balance in language agent scenarios. We have also noted your concerns about our technical contribution and comparison strategies in our work. We hope to address these concerns below: **Weakness 1:** *Limited technical contribution* **Response:** Besides the technical contribution of demonstrating our training procedure, We would also like to highlight our research value and community benefits, including research study and insights to guide future language agent development, establishing comprehensive language agent evaluation, and open-sourcing models with superior performance for the language agent community. We are very grateful to `Ycpy`, `JHRb`, and `QCzC` for mentioning these contributions. - **Research insights on language agent development:** Our insights into harmonious abilities motivated us to develop Lemur models, and our experiments indicate its direct importance to language agents compared to unbalanced models like Llama or CodeLlama. To provide more insights into language agent development, We have further added more human analysis in the appendix of the revised version, indicating the direction to further develop open language agents. - **Comprehensive language agent research study:** Previous works usually studied language agents with specific tools and environments. In our work, we establish a comprehensive language agent study by organizing diverse environments to evaluate key agent skills in different scenarios. We are glad that this is recognized by all reviewers. - **Open-sourcing effort for language agents community as a key technical contribution:** Current AI agent research largely relies on prompting close-sourced models like GPT-4, GPT-3.5 Turbo, and Claude-2. There are a lot of limitations when accessing close-sourced models by APIs, such as high costs and difficulties in custom training. We made expansive efforts to develop and open-source Lemur models. We believe that it will help the AI agent community to develop and explore LLM-based agents by not only prompting models but also conducting agent training with different learning methods like RL. By being openly available, Lemur models can be accessed by individuals and organizations that might not have the resources to develop their own models from scratch, democratizing access to agent models. --- **Weakness 2:** *Unequal Model Size Comparison on the code benchmark* **Response:** Thanks for your detailed feedback on the model comparison! - To better track the performance of our model on code benchmark, we selected CodeLlama-34B and CodeLlama-34b-Instruct as representative baselines, which were the largest and most capable open-sourced code models. - We would also like to highlight that our motivation for training the Lemur models is to harmonize the model’s natural language and code capabilities for agent tasks. Being the most capable model on code benchmarks may not be our focus. Instead, we may emphasize the model’s overall performance in both natural language, code, and agent tasks. --- **Question 1:** *Reason for 10:1 Code-to-Text Ratio in Pretraining* **Response:** - The ratio of code to text is a complex trade-off between the model size, the vanilla performance of the model, data quality, learning efficiency, catastrophic forgetting, etc. - To roughly estimate a good ratio of composing the training corpora, we conducted a continue pre-training study with Llama models on a set of different code-text ratios and found that the ratio of 10:1 is an efficient ratio for the Llama model to transfer from text to text-code balance. - Due to computational limits, it is difficult for us to conduct comprehensive experiments and perform systematic studies on this scale. However, we hope that our open-sourced effective settings and training checkpoints can benefit the community for continual exploration. - We believe that a method to predict the optimal continue-pre-training data mixture ratio for a pair of domains to maximize their performance would be very meaningful and interesting, and it is still an open research question for the current large language model. We have added this explanation in Appendix A of our revised version. Thanks for your valuable question!

Response to reviewer Ycpy

Thank you for recognizing our work! We are delighted to hear your appreciation for our training methods in designing and building language agents, as well as the recognition of our detailed study on language agent abilities. At the same time, we are happy to further explain the following topics regarding our motivation and the questions you have: --- **Weakness 1:** *Ambiguous Motivation for Integrating Programming Language Skills in Language Agents* **Response:** Thank you for your feedback regarding the motivation in our paper! We would like to further clarify our motivation: - The most significant difference between agents and chatbots is that agents not only need to communicate with human users using natural language, but also need to interact with complex environments using executable programming languages. For example, the agent of web browsing accepts user instructions and controls the browser using code to complete tasks in multiple steps. - Therefore, this motivates us to directly enhance the code capabilities to help agents manipulate the environment via code. We would like to show the direct importance of programming language in agent scenarios, rather than the relevance or benefits of programming languages to natural languages. --- **Question 1:** *Choice of Scripting Languages for Language Agents* **Response:** - In section 2.1, we briefly mentioned that, among all languages, we focus on scripting or interpreted languages (Python, SQL, Bash, Perl, etc.) because agent models are often executed in interactive scenarios. Unlike compiled languages like C++, the interpreted nature of scripting languages allows immediate execution and easy modification, which is essential for dynamic interaction in language agent scenarios. - Another interesting finding is that, although the training corpora of Lemur contain a large proportion of scripting languages, we observed a general performance improvement on mainstream programming languages compared to the original LLaMa-2. The following are the detailed results of MultiPL-E in Table 3, which is a multi-programming language evaluation benchmark. | Model | C++ | Java | PHP | TypeScript | C# | Bash | Average | | --- | --- | --- | --- | --- | --- | --- | --- | | LLaMA-2-70B | 30.40 | 31.70 | 34.20 | 15.10 | 25.90 | 8.90 | 24.4 | | Lemur-70b-v1 | 34.78 | 35.40 | 35.40 | 35.40 | 25.47 | 15.82 | 30.4 | --- **Question 2&3:** *Replicability for those models other than Llama or smaller than 70B.* **Response:** - Harmonizing model abilities by data mixture is a general and effective approach. For those base models other than Llama or smaller than 70B, we believe that it is generally capable of harmonizing natural language and code capabilities. - When generalizing this approach to different models, the data mixture ratio and training steps need further adjustment. For example, smaller models have less capacity for harmonizing both text and code capabilities. Therefore, they may suffer from relatively obvious forgetting of natural language knowledge. The strategy of data mixture for large language model pretraining is an open and valuable research question. We believe we will have a more efficient and predictable data mixture strategy in future studies. --- We hope our explanation could address your concern! We further revised our paper and appendix to clarify our motivation, incorporating your valuable inputs to better articulate our rationale.