Rigorous software testing is crucial for developing and maintaining high-quality code, making automated test generation a promising avenue for both improving software quality and boosting the effectiveness of code generation methods. However, while code generation with Large Language Models (LLMs) is an extraordinarily active research area, test generation remains relatively unexplored. We address this gap and investigate the capability of LLM-based Code Agents to formalize user issues into test cases. To this end, we propose a novel benchmark based on popular GitHub repositories, containing real-world issues, ground-truth bug-fixes, and golden tests. We find that LLMs generally perform surprisingly well at generating relevant test cases, with Code Agents designed for code repair exceeding the performance of systems designed specifically for test generation. Further, as test generation is a similar but more structured task than code generation, it allows for a more fine-grained analysis using issue reproduction rate and coverage changes, providing a dual metric for analyzing systems designed for code repair. Finally, we find that generated tests are an effective filter for proposed code fixes, doubling the precision of SWE-Agent. We release all data and code at https://github.com/logic-star-ai/SWT-Bench
Paper
Similar papers
Peer review
Summary
In this paper, the authors propose a novel benchmark for evaluating the capability of LLMs and Code Agents in generating test cases from GitHub issues. The benchmark includes real-world issues, patches, and golden tests from popular Python repositories. The authors claim that Code Agents outperform traditional test generation methods and highlight the potential of these agents in improving software quality and developer productivity.
Strengths
The paper proposes a new benchmark for test generation, addressing a gap in current research. The approach leverages real-world data from GitHub, making the benchmark relevant and practical.
Weaknesses
The data contamination problem. The proposed benchmark is collected from GitHub, which might have been exposed to the pretraining corpus of LLMs. The authors overlook substantial existing work on test case generation using LLMs. There is no discussion on the long-term relevance and maintenance of the benchmark, which is crucial for its sustained utility. The benchmark is focused solely on Python, ignoring other programming languages. The experiments are conducted on only three LLMs (GPT-4, Claude-3, Mixtral 7x22B), which is insufficient for a comprehensive evaluation of the benchmark. The paper fails to address the broader societal impact.
Questions
How does the benchmark ensure that it does not include tasks already present in pre-trained DNN datasets? Why were significant existing works on LLM-based test generation not included in the paper? What measures are in place to ensure the long-term relevance and maintenance of the SWT-BENCH benchmark? How might the results differ if the benchmark included tasks in programming languages other than Python? Can the authors discuss the potential societal impacts of the widespread adoption of Code Agents in software testing, particularly concerning developer employment? Please see the above to clarify any misunderstandings and add additional results.
Rating
3
Confidence
4
Soundness
2
Presentation
1
Contribution
2
Limitations
The paper lacks discussion on various limitations, including the limitations of the metrics used and potential data contamination across models.
Response by Reviewer ueFW
Thank you for the responses, and sorry for the late response due to a flurry of proposals and review dues. I have increased my overall assessment as some of my concerns have been addressed. While others are shown in below. **Data contamination** Thanks for your experiments for GPT-3-preview 1106, the experiments do not address my concern. At least, I need to mention the data contamination for all models by directly providing previous inference results and dividing them based on the KC time. **Existing works not included in the paper?** Thanks for your response. But I need to mention that omitting CodeT is crazy. **Other languages?** The response does not address my concern. **Other LLMs** I would recommend providing experiments for OpenCodeInterpreter, DeepSeek-Coder, XwinCoder, CodeLlama, WizardCoder, and Starcoder2 family in open-source LLMs for a new benchmark.
Reply to Reply to ueFW
We thank the reviewer for acknowledging our rebuttal, engaging in the discussion, and raising their score. We would like to address their comments below. **Can you provide the data contamination results for each model?** Below we report results on SWT-bench Lite for all considered models, split by whether the issue was created before or after the corresponding Knowledge Cutoff (KC). The results correspond to Table 2 in the global attached PDF, comparing the model performances using SWE-Agent. Note that for our previous results on GPT4, we considered the full SWT-bench to ensure a sufficiently large number of samples from after the KC is available. Due to time constraints, this is not possible here and the smaller size of SWT-bench Lite results in much fewer instances being available after the KC. In fact, many of the recent models have a KC later than the latest instance (15 Aug 2023) in the full SWT-Bench (see the table below). For the only model with sufficiently early KC (Mixtral 8x22B), the overall F2P rate is so low that no meaningful comparison is possible (2/190 F2P instances before KC and 0/89 after). | Model | KC | PR created | n | Applicable |F2P | Coverage | |---------------------------|-------------|--------------|-----|--------------|-------|------------| | GPT-4 Preview 1106 | 30 Apr 2023 | before KC | 268 | 76.9 | 16.4 | 18.1 | | | | after KC | 11 | 54.5 | 18.2 | 20.0 | | Mistral Large 2 | 31 Jan 2024 | before KC | 279 | 4.3 | 0.7 | 0.2 | | | | after KC | 0 | - | - | - | | Claude 3.5 Sonnet | 30 Apr 2024 | before KC | 279 | 60.2 | 12.2 | 22.1 | | | | after KC | 0 | - | - | - | | GPT-4o mini (2024-07-18) | 31 Oct 2023 | before KC | 279 | 53.8 | 9.7 | 13.0 | | | | after KC | 0 | - | - | - | | Claude 3.0 Haiku | 31 Aug 2023 | before KC | 279 | 6.8 | 2.9 | 1.9 | | | | after KC | 0 | - | - | - | | Mixtral 8x22B | 30 Sep 2021 | before KC | 190 | 2.6 | 1.1 | 0.0 | | | | after KC | 89 | 2.2 | 0.0 | 1.4 | **Why did you not discuss CodeT in the paper?** As discussed in detail in our response to Reviewer PhkB, CodeT is not applicable to our setting, and we thus omitted it in our literature review due to space constraints. However, we will make sure to include it with a suitable discussion in the next revision of our paper, using the extra space of the camera ready version. **Why did you not consider popular Open-Source Code Models like CodeLlama?** During our experimentation, we indeed evaluated CodeLlama-70B and WizardCoder-Python-34B. However, we found they were not capable of following the desired diff format or agent instructions. Instead, they produced incoherent, incorrect, and degenerated outputs. As they thus consistently yielded close-to-zero F2P rates, we have excluded them from the reported results. We do not believe that comparing such low F2P rates would be interesting or meaningful, but are happy to include results for these models in the appendix of a revised version. Since both the strongest models (e.g. GPT4) alone (ZeroShotPlus) and slightly weaker models (e.g. Claude 3.0 Haiku) in an agent framework struggle with SWT-Bench, we believe it is most interesting to focus on comparing different agent frameworks using the best available base models. **Conclusion** We hope to have been able to address the reviewers concerns and are looking forward to their response.
Dear authors Thanks for your response. I find that the Applicable (pre-KC) for GPT-4-turbo-preview are largely different with post-KC. Can you clarify that?
Reply to Reply to Reply to ueFW
We thank the reviewer for their quick reply and hope that we could successfully address all points they did not have follow-up question on. **Effect of Data Contamination on Applicability for GPT4** While the applicability is indeed higher for issues created before the KC for GPT4, both F2P rate and Coverage are lower, which are much stronger indicators for memorization. Applicability solely measures whether the LLM can respect the required diff format, while F2P and Coverage measure the correctness of the generated tests. While memorization would lead to a correct test (higher F2P and coverage) it would not necessarily improve applicability as the test would not have been included in the training data in the right diff format. Further, we want to highlight that only 11 samples after the KC were available, making any conclusions based on these results statistically questionable (as highlighted in our previous reply).
Summary
This paper introduces SWT-BENCH, a novel benchmark for evaluating automated test generation capabilities of AI models, particularly Code Agents. The authors adapt existing code repair datasets and methods to the task of test generation, proposing new metrics such as fail-to-pass rate and change coverage. Their experiments reveal that Code Agents, originally designed for code repair, outperform methods specifically designed for test generation. The best-performing method, SWE-AGENT+, achieves an 11.1% success rate in generating relevant fail-to-pass tests. The study also demonstrates significant complementarity between different approaches, with an ensemble of methods solving 70% more samples than the best single method. Additionally, the authors show that generated tests can serve as a strong signal for the correctness of proposed code fixes. While the study is limited to Python and may have some selection biases, it suggests that Code Agents have significant potential for automated test generation, potentially improving both software quality and developer productivity.
Strengths
Originality: - Introduces SWT-BENCH, a novel benchmark for test generation, adapting existing code repair datasets to a new task. - Proposes the use of Code Agents for test generation. - Develops new metrics (fail-to-pass rate and change coverage) specifically for evaluating test generation. Quality: - Comprehensive evaluation of multiple methods, including baselines and state-of-the-art approaches. - Rigorous experimental setup with detailed reporting of methodologies and results. - Great analysis of results, including complementarity between methods and correlation with code repair performance. Clarity: - Well-structured paper with clear explanations of complex concepts. - Effective use of figures and tables to illustrate key points and results. - Detailed appendices providing full prompts and additional experimental details. Significance: - Demonstrates the potential of Code Agents for automated test generation, a critical area in software development. - Shows that generated tests can effectively validate code fixes, potentially improving code quality processes. - Provides a new benchmark (SWT-BENCH) that could drive further research in this area. - Highlights the complementarity of different approaches, suggesting potential for ensemble methods in this domain.
Weaknesses
1. Limited statistical analysis: The authors acknowledge they were unable to perform a full statistical analysis due to computational costs. This limits the confidence in the reported results. It would be great if they can conduct a power analysis to determine the minimum number of runs needed for statistical significance. 2. Lack of error analysis: The paper doesn't provide a detailed analysis of the types of errors made by different methods. Maybe they should categorize and quantify common error types for each method. Provide qualitative examples of generated tests, both successful and unsuccessful. Analyze how error types correlate with issue complexity or repository characteristics. 3. Limited exploration of hyperparameters: The paper doesn't discuss the impact of different hyperparameters on the performance of Code Agents. To improve: Conduct an ablation study on key hyperparameters (e.g., number of interaction rounds, temperature). Provide insights on how to optimize Code Agents specifically for test generation. 4. Insufficient comparison to human performance: The paper lacks a comparison to human-generated tests. To address this: Include a small-scale study with professional developers generating tests for a subset of issues. Compare the quality, coverage, and time efficiency of AI-generated vs. human-generated tests. 5. Narrow focus on Python: The study is limited to Python, potentially limiting generalizability.
Questions
1. Given the computational constraints that prevented a full statistical analysis, could you provide more details on the variability of your results? Even with limited runs, could you estimate confidence intervals for your main findings? 2. The paper lacks a detailed error analysis. Could you provide examples of common failure modes for the different methods, particularly for SWE-AGENT+? How do these failure modes relate to the characteristics of the issues or repositories? 3. How sensitive are the Code Agents' performances to different hyperparameters? Did you explore variations in the number of interaction rounds or temperature settings? If so, what insights can you share about optimizing Code Agents specifically for test generation? 4. The paper doesn't compare AI-generated tests to human-generated ones. Have you considered conducting even a small-scale comparison with professional developers? This could provide valuable context for understanding the practical impact of your methods.
Rating
5
Confidence
4
Soundness
3
Presentation
3
Contribution
2
Limitations
1. Limited Statistical Analysis: Due to computational constraints, the authors were unable to perform a full statistical analysis. This limits the confidence in the reported results and makes it difficult to assess the robustness and reproducibility of the findings. 2. Lack of Detailed Error Analysis: The paper doesn't provide an in-depth analysis of the types of errors made by different methods. This limits understanding of where and why the methods fail, which could be crucial for further improvements. 3. Python-Centric Approach: The study focuses exclusively on Python, which may limit the generalizability of the results to other programming languages. 4. Absence of Human Baseline: There's no comparison between AI-generated tests and human-generated tests, making it challenging to assess the practical impact and efficiency gains of these methods in real-world scenarios. 5. Limited Hyperparameter Exploration: The paper doesn't thoroughly explore the impact of different hyperparameters on the performance of Code Agents, potentially missing opportunities for optimization. 6. Lack of Runtime Performance Analysis: While computational costs are mentioned, there's no detailed analysis of execution times for different methods, making it difficult to assess practical trade-offs.
Summary
This paper focuses on automatic test generation using large language models (LLMs) and code agents. The authors introduce a benchmark called SWT-BENCH, which aims to analyze the performance of LLMs and agents in generating unit tests given a task description. The key contributions include: 1) Creating the SWT-BENCH testing benchmark, which contains over 1700 samples from real-world GitHub issues. 2) Benchmarking various LLMs and agent systems. This includes evaluating direct LLM prompting, existing test generation tools (e.g., LIBRO), and adapted code agents (e.g., SWE-AGENT, AUTOCODEROVER). 3) Introducing a new format for LLM-based code editing.
Strengths
1) LLMs are stochastic black boxes by nature, and building experimental testbeds is crucial to advance the field. They can provide training data for the models as well as relevant signals for improvements. I think this benchmark is a very valuable contribution to the field as it will enable more algorithmic and agentic advances on the topic. 2) This benchmark does a good job at testing several methods such as LLMs, agents, and modifications to existing agentic systems. 3) The performance analysis of various methods is quite detailed, and the correlation or lack thereof between methods is very interesting.
Weaknesses
1) The scope of the dataset could be expanded further. In its current form, it only addresses Python codebases with associated GitHub PR issues. The distribution of testing problems encountered by real-world software engineers is likely quite different from those represented in this testbed. 2) A private test set that is not part of the publicly available training data would strengthen the claims that could be made when improvements are observed on this benchmark.
Questions
1) Have you looked into leveraging the failure modes of the generated test cases to improve code repair or code generation tasks? 2) LLMs for code generation seem to be very stochastic by nature. You mention that the failure modes on the benchmark vary depending on the underlying method used. To what extent do you think this variation is explained simply by the stochasticity of the method? For example, if you ran swe-bench 5 times, would that variation disappear? 3) How do you think would your method generalize in a private codebase that isn't available online?
Rating
8
Confidence
4
Soundness
4
Presentation
4
Contribution
4
Limitations
The contamination problem with LLMs is, in my opinion, very important and under-discussed. Due to GitHub data being public, a lot of the data already exists in the training data for LLMs, which can render some conclusions less impressive. This is especially true in the code repair and test generation areas. I think it would be very helpful to include some analysis of the contamination problem. This could be done by using a held-out dataset in a similar way to how the benchmark below does it: https://livecodebench.github.io/
Summary
This paper re-purposes SWE-Bench, a previous benchmark on repository-level code generation, to SWT-Bench, a benchmark for test generation by asking LLMs to generate tests to reproduce the issues for various GitHub repos and see if such tests can capture the bugs before the gold-standard patches are applied and whether the generated tests will pass after the patches are applied (i.e., Fail -> Pass). Experiments are conducted with multiple baselines and results show that SWE-Agent, the previously proposed method for code repair, achieves the best performance for test generation.
Strengths
S1. This paper targets on test generation, which is a relatively under-explored area and could be useful for increasing popular coding agents; S2. This work thoroughly studies the relation between the tasks of code repair and test generation, and show how we can turn a code repair dataset into a test generation dataset using SWE-Bench -> SWT-Bench as an example; S3. The designed metrics to evaluate the generated tests (i.e., Fail-to-Pass rate, Change Coverage, and Patch Applicability) are quite interesting and reasonable, which could be useful for further work on automatic test generation evaluation.
Weaknesses
I think the main weakness of this paper is over-claiming. While it is a clever idea to re-purpose code repair benchmarks as SWE-Bench into a test generation benchmark, there are quite a lot limitations of this method for the conclusions on the resulting dataset to claim "state of the art software tester". The main purpose of software testing is not to reproduce the issues, but given the specifications, crafting inputs and the constraints on the outputs to makes sure that the code to be tested has the expected behavior given different inputs. To this end, I believe there is a large gap between the ability that SWT-Bench is testing, and the actual test generation ability needed for the job of software testers. To make such a claim, I believe at least a few software testing benchmarks (e.g., https://swtesting.techconf.org) need to be used and the results to be reported.
Questions
Please respond to the concerns in the "Weakness" section.
Rating
4
Confidence
4
Soundness
3
Presentation
3
Contribution
2
Limitations
N/A
Thanks for the response
I'd like to thank the authors for the response. > While our title may be placative, it only implies that Code Agents perform better than other automated systems on the studied task. I think this is where the major discrepancy is: the studied task is **GitHub issue reproduction**, which is quite far from what people would normally think of the work of **software testers**. Thus I still think the title is over-claiming and somewhat misleading, and I'm afraid I could not give higher score for this work. Respectfully, Reviewer i5fN
Reply to Reply of i5fN
We thank the reviewer for their response and engaging in the discussion. We are happy to modify the title of our submission (in accordance with the submission guidelines) to follow the reviewer’s suggestion and avoid over-claiming. We propose to adjust the title to **Can Code Agents Reproduce Real-World GitHub Issues?**. As promised in the initial rebuttal, we will also revise the text to make the scope of our work more explicit. We are happy to answer any followup questions or suggestions they might have.
Summary
Authors present a new benchmark for the SWE tasks of generating tests corresponding to an issue in a codebase with unit tests. They propose to repurpose the SWE-bench dataset for this task. They also evaluate LLM based prompting and agentic approaches (SWE-Agent and AutoCodeRover) with metrics like code change coverage and fail to pass rate that are formally defined and measured.
Strengths
- This work focuses on generating test cases corresponding to user issues with a codebase with existing unit tests. This problem has been not studied in such detail in my knowledge. Authors have presented this problem with sufficient motivation. - While authors adapt an existing code fix benchmark (SWE-bench) for their task (SWT-bench), their contributions in the form of task formulation, corresponding new metrics and exhaustive analysis of the benchmark could be key to future research on this topic. - Authors also adapt popular LLM approaches like SWE-Agent and ACR for their benchmark and show their effectiveness on generating tests. The demonstrated impact automated test generation has on automated code fixing is particularly promising. Their results include exhaustive analysis of the performance of different methods on the proposed benchmark. - The paper is very well written and clear to follow.
Weaknesses
- Arguably the most significant impact of automatic test generation would be on automated code repair/generation. While authors describe this aspect of their work (Lines 328-334), providing more details and large-scale experiments could significantly strengthen the perceived utility of this work. - CodeT by Chen et al https://arxiv.org/abs/2207.10397 is a key related work that was missed in the related work.
Questions
- Lines 328-334: You mention about the impact on precision and recall, can you share more details of the method used and the final correctness rate with synthetic tests generated by the Agent? - What are some future directions you would recommend to improve the performance of LLM Agents on SWT-bench and SWE-bench based on your study? A future work recommendation would greatly help the completeness of the paper.
Rating
7
Confidence
4
Soundness
3
Presentation
4
Contribution
3
Limitations
Authors have briefly discussed limitations of their work in the main paper.
Author-Reviewer discussion (Aug 7 - Aug 13)
Dear Submission15335 reviewers, We appreciate your reviews as we move into the Author-Reviewer discussions phase (Aug 7 - Aug 13). Please read all reviews and author responses carefully. Please address any remaining questions or concerns with the authors and respond to their rebuttal as needed. Authors need time to respond to your messages, so please post your responses as soon as possible, so there is time for back-and-forth discussion with the authors. At a minimum, please acknowledge that you have read the author rebuttal. Based on the material provided, please adjust your scores and reviews as necessary. Dear Submission15335 authors, We appreciate your rebuttal. Please continue the dialogue with the reviewers during this discussion phase. This message and thread are visible to both reviewers and authors. If you need to respond privately, adjust the "Readers" setting accordingly.
Questions for authors
Dear authors, Thank you for your rebuttal. I have couple questions about your submission that may clarify certain aspects of your paper and facilitate the discussion with reviewers. Page 3: "Golden Tests" -> are these tests that were added in the patch under consideration? Page 4: "If none of the lines modified by the golden patch X are executed by any test, i.e., |X ∗r | + |X ∗166 a | = 0, we exclude this instance from our coverage analysis (43.1% of the cases)." - can you elaborate on this? How does the patch fix an issue if none of the tests execute any lines in it? Is the patch deleting erroneous code? 43% seems like a large percentage of cases for such occurrences. Page 7, table 2: What is "valid patch" in section 5.2? Is it a test case patch that is valid? You are not talking about a code patch that fixes the issue, right? How is the validity checked? Maybe consider renaming the "valid patch", since it might be misleading for readers when test change is named "patch" the same way as code fix change is named "patch". What is fail-to-any tests (F→×)? Why are they considered to be good? Can you elaborate? Isn’t there a risk that LLM generates a meaningless test that fails both on initial codebase and on the patched codebase and therefore does not contribute a valid test? Thank you
Response to the AC
We thank the AC for taking such an active role in chairing the discussion and are happy to answer their questions below: **What are golden tests?** Exactly as the AC suggests, these are the tests that were added in the pull request that “resolved” the GitHub issue underlying the SWT-Bench instance. At least one of these tests fails on the original state of the codebase, before the golden code patch is applied, and passes after. They are thus always a valid (gold standard) solution for the SWT-Bench task. We will make this more clear in Line 32 where they are first introduced. **Can you discuss what can cause none of the lines modified by the golden code patch to be covered by the test suite?** The main reason for this is that Python’s _trace_ module by default is unable to measure coverage in subprocesses. If (part of) a test suite is executed using such independent processes, coverage can not be measured accurately and no execution of the patched code is reported. Meanwhile, threading based concurrency is handled correctly. We would like to highlight that this issue comes from the task instance and Python’s _trace_ module, but not the test generation method. The measured coverage is thus still comparable across different test generation methods and computed over the remaining 57% of SWT-Bench instances. **Can you clarify what a _valid_ patch is in Section 5.2?** A valid patch is one that can be successfully/without error applied to a codebase, i.e. that has the correct format. As generated tests are added to the code base in the form of a patch, we reused the term “valid patch”, currently defined implicitly in Line 170. We will follow the AC’s suggestion and rename them to “well-formed tests” and introduce this terminology explicitly to improve clarity. As many models struggle to produce valid/well-formed (i.e. applicable without error) test patches in the default unidiff standard, the rate of valid test patches is a relevant metric in particular when comparing our novel (ZeroShotPlus) with the standard unidiff patch format (ZeroShot). **What is a fail-to-any (F -> X) test and why is it “good” to measure this?** A fail-to-any test is one that fails on the original codebase and either passes or fails on the codebase with the golden patch applied. While some F->X tests might be undesirable as the AC suggests, we consider the rate of F->X tests as a useful metric for assessing a test agent’s overall capabilities for two main reasons. First, creating an F->X test is a prerequisite for generating a fail-to-pass (F->P) test, which is ultimately the desired outcome. Therefore, the rate of F->X serves as an intermediate metric for our final F->P metric. Second, given that the agent has access to the original version of the codebase and the issue report, it can leverage these two inputs to self-verify if a test fails on the codebase before proposing the test. The rate of F->X tests reflects the agent’s ability to perform this self-verification. We will add this discussion in the corresponding section. **Conclusion** We hope to have been able to address the AC’s questions and remain happy to answer any follow-up they might have.
Reviewer responses to authors (Aug 7 - Aug 13)
Dear Submission15335 reviewers, We appreciate your reviews. We have only two days left for the author-reviewer discussions. Please read all reviews and author responses. It is helpful if you respond to authors' rebuttal and address any remaining questions or concerns with the authors. At a minimum, please acknowledge that you have read the author rebuttal. If your evaluation of the paper has changed, please adjust your scores and reviews as necessary. Thank you.
Decision
Accept (poster)