MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution

Summary

This paper designs a Multi-Agent framework, MAGIS. Following the form of team problem solving, it designs four agents: Manager, Repository Custodian, Developer, and QA Engineer, which are used to decompose tasks, locate problem codes, generate codes, and review codes, respectively. They help the LLM to better locate problems and generate correct codes in a cooperative way. This framework can solve 13.94% of problems on the SWE-bench benchmark.

Strengths

1. This paper draws on the cooperation method of human team, assigns different roles to agents, and utilizes the powerful understanding ability of the big model for natural language, as well as the professional ability in specific fields stimulated by being assigned specific roles. Compared with simply calling the big model, the effect is improved. 2. This paper draws on the software engineering idea very well. For example, the Kick-off Meeting method is adopted, which can help multiple developers clarify independent tasks, ensure that there is no conflict, and can determine the order of modification.

Weaknesses

1. The paper uses 11 prompts in total, and the specific prompt content cannot be found in the main text, which makes people worry whether the experimental results will strongly depend on the quality of the designed prompts. After replacing other models, the effect may be uncontrollable. 2. The framework is relatively complex and requires a large number of interactions with LLMs. In the end, compared with other baseline scores, whether these additional computing resources are necessary is not further analyzed in the paper. 3. **SWE-bench clearly states that the hints field should not be used in the submission list, but from this paper, it is found that hints seem to be the key to MAGIS improvement.** Is this unfair to other methods? 4. I noticed that there are a lot of new submissions on SWE-bench (Lite) recently. Do you need to do a new comparison? (Of course, I think it is reasonable not to compare, after all, they are working in the same period.) ### Reference - [1] Opendevin: Code less, make more. https://github.com/OpenDevin/OpenDevin/, 2024. - [2] Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, Anton Cheshkov, Jun Sun, Hao Yu, Guoliang Dong, Artem Aliev, et al. Coder: Issue resolving with multi-agent and task graphs. arXiv preprint arXiv:2406.01304, 2024. - [3] Yingwei Ma, Qingping Yang, Rongyu Cao, Binhua Li, Fei Huang, and Yongbin Li. How to understand whole software repository? arXiv preprint arXiv:2406.01422, 2024. - [4] Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. Autocoderover: Autonomous program improvement, 2024.

Questions

See the questions in Weaknesses.

Rating

4

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors explain the limitations of their approach in the paper.

Summary

The paper studies the reasons behind LLMs' failures in resolving GitHub issues, identifying key factors such as locating code files and lines, and the complexity of code changes. The authors propose a novel multi-agent framework, MAGIS, comprising four specialized agents: *Manager* and *Repository Custodian* for planning, *Developer* for coding, and *Quality Assurance Engineer* for code review. Experimental results show that MAGIS significantly outperforms GPT-4 and Claude-2, solving 13.94% on the 25% subset of the SWE-Bench dataset. Ablation studies confirm the effectiveness of each agent's role in the framework.

Strengths

- The paper tackles the challenging problem of using LLMs to resolve GitHub issues. The proposed method is intuitive and effective, achieving an eight-fold performance gain, compared to baseline GPT-4. - The paper conducts massive experiments on both the reason why LLMs fail in the process of resolving GitHub issues, and the role of each agent, providing insights for the research community.

Weaknesses

- The reproducibility of experiments for baseline models like GPT-4 and Claude is unclear. The code and data are temporarily unavailable, hindering follow-up and comparison by other researchers. - Typos: There are minor typos, such as "multi-agennt" instead of "multi-agent" on page 2, line 63, and some missing "%" signs in the resolved ratio on page 7.

Questions

1. In Section 2, I'm a bit confused about the calculation of coverage ratio. How is $[s_i, e_i]$ defined for pure 'deleting' and 'adding' operations? If many code adjustments occur in a single file, does it ensure that each adjustment contributes independently to the coverage? I mean, if model generated code is different from the reference code, the position behind is not aligned.) 2. In Section 4, can you kindly provide more key implementing details, especially for the model comparison experiment in Table 2? Some helpful information may be the core instruction/input for each model, the number of interaction rounds, and whether code interpreter is allowed. Those are all related to the performance of the model. 3. Do you submit (or plan to submit) the results to SWE-Bench official leaderboard? How do this method differ from other agent-based methods?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, the authors adequately addressed the limitations.

Summary

This paper introduces MAGIS, a novel Large Language Model (LLM)-based multi-agent framework designed to address the challenge of resolving GitHub issues in software development. The authors conduct an empirical study to identify key factors affecting LLMs' performance in this task, including file and line localization accuracy and code change complexity. Based on these insights, they propose a collaborative framework consisting of four specialized agents: Manager, Repository Custodian, Developer, and Quality Assurance Engineer. These agents work together through planning and coding phases to generate appropriate code changes. The framework is evaluated on the SWE-bench dataset, demonstrating significant improvements over existing LLMs in resolving GitHub issues. Specifically, MAGIS achieves a resolved ratio of 13.94%, which is an eight-fold increase compared to the direct application of GPT-4. The paper also provides detailed analyses of the framework's components and discusses its effectiveness in different scenarios.

Strengths

Originality: 1. The paper presents a in-depth empirical analysis of LLMs' performance in resolving GitHub issues, providing unique insights into the challenges of applying AI to complex software engineering tasks. 2. The study leverage software engineering metrics, such as the line locating coverage ratio, to quantify LLM performance in code change generation. Quality: 1. The empirical study demonstrates rigorous methodology, examining multiple factors affecting LLM performance including file localization, line localization, and code change complexity. 2. The analysis employs statistical techniques to establish correlations between various complexity indices and issue resolution success, adding depth and reliability to the findings. Clarity: 1. The paper clearly articulates the gap between LLMs' performance on function-level tasks versus repository-level tasks, providing a strong motivation for the study. 2. The authors present their empirical findings with well-designed visualizations and tables, making complex data easily interpretable. Significance: 1. The empirical analysis provides crucial insights that bridge theoretical understanding of LLMs with practical software engineering challenges. 2. The findings directly inform the design of more effective AI-assisted software development tools, as demonstrated by the MAGIS framework. 3. The study lays a foundation for future research in applying AI to software engineering tasks, potentially influencing the direction of both AI and software engineering fields.

Weaknesses

1. Limited dataset diversity: The study relies solely on the SWE-bench dataset, which, while comprehensive, is limited to 12 Python repositories. This may not fully represent the diversity of real-world software projects. 2. Computational resources and efficiency: The paper doesn't discuss the computational requirements or efficiency of MAGIS compared to simpler approaches. 3. Potential overfitting to SWE-bench: The framework might be tailored too specifically to perform well on SWE-bench, potentially limiting its generalizability. 4. Lack of user study: The paper doesn't include feedback from actual software developers on the usefulness and quality of MAGIS's solutions. 5. Limited discussion on prompt engineering: While the paper mentions using prompts, it doesn't delve into the specifics of prompt design or its impact on performance.

Questions

1. Your study focuses on the SWE-bench dataset, which is limited to 12 Python repositories. Have you considered how MAGIS might perform on a more diverse set of programming languages and project types? What steps could be taken to validate the framework's generalizability beyond Python projects? 2. The paper doesn't discuss the computational requirements of MAGIS. Can you provide information on the runtime and resource requirements of MAGIS compared to simpler approaches? How does this impact its practical applicability in real-world software development environments? 3. How have you ensured that MAGIS is not overfitting to the specific characteristics of the SWE-bench dataset? Have you tested the framework on any GitHub issues outside of this dataset to validate its real-world applicability? 4. While you describe the roles of different agents, there's limited analysis of how their interactions contribute to overall performance. Have you conducted any ablation studies on different interaction patterns between agents? This could provide insights into which aspects of the multi-agent approach are most crucial for success. 5. While you mention using prompts, there's limited discussion on prompt design. Could you provide more details on your prompt engineering strategies and how they impact the framework's performance?

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

1. The authors acknowledge in Appendix K that the SWE-bench dataset, while representative, may not fully reflect the diversity of all code repositories, particularly in specialized fields or different programming paradigms. However, SWE-bench is known as one of the most challenging benchmarks, this might be ok. 2. In Appendix K, the authors mention the potential impact of prompt design on LLM performance and the difficulty in eliminating prompt bias completely. 3. The authors note in Appendix K that applying their findings to other code repositories may require further validation due to the limited sample scope.

Summary

This paper proposes MAGIS, a multi-agent coding framework for solving patch generation tasks. The roles consist of Manager, Repository Custodian, Developer and QA engineer, with the task of planning, file location, file editing, and review, respectively. Experiments on the SWE-bench benchmark show that performance is improved over baselines.

Strengths

- The proposed multiagent framework is simple and effective - The proposed model is effective with and without using the hints provided in SWE-bench - Ablations demonstrate the effectiveness of the QA engineer role

Weaknesses

My concerns are along the evaluation and comparisons: - The main experiments are limited, as only GPT-4 is measured - Baselines such as SWE-agent are missing

Questions

- SWE-bench Lite results (appendix D) should be in the body of the paper - Is ablation of the kickoff meeting possible? This is an interesting mechanism but it's hard to understand whether it is effective or not - Can you explain/show more about the prompts and exemplars (if exemplars are used) for each role?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

yes

Request for Feedback Before Discussion Deadline

Dear Reviewers and Area Chair,   As the discussion deadline approaches, we kindly request all reviewers to review our responses to your comments and share any remaining concerns or questions. We highly value your feedback and want to ensure that all issues are fully addressed before the final decision. We would also like to specifically thank reviewer 1wgB for the thorough discussion and valuable feedback provided so far. We have carefully responded to all of the concerns raised during the review and discussion phases by adding additional experiments and explanations. We hope that these clarifications might lead to a reconsideration of the current score. We sincerely appreciate your valuable time and effort in reviewing our submission.   Best regards, The authors

Response to authors

Thank you to the authors for the detailed response. The response addressed my concerns.