Large language models of code (Code-LLMs) have recently brought tremendous advances to code completion, a fundamental feature of programming assistance and code intelligence. However, most existing works ignore the possible presence of bugs in the code context for generation, which are inevitable in software development. Therefore, we introduce and study the buggy-code completion problem, inspired by the realistic scenario of real-time code suggestion where the code context contains potential bugs -- anti-patterns that can become bugs in the completed program. To systematically study the task, we introduce two datasets: one with synthetic bugs derived from semantics-altering operator changes (buggy-HumanEval) and one with realistic bugs derived from user submissions to coding problems (buggy-FixEval). We find that the presence of potential bugs significantly degrades the generation performance of the high-performing Code-LLMs. For instance, the passing rates of CODEGEN-2B-MONO on test cases of buggy-HumanEval drop more than 50% given a single potential bug in the context. Finally, we investigate several post-hoc methods for mitigating the adverse effect of potential bugs and find that there remains a significant gap in post-mitigation performance.
Paper
Similar papers
Peer review
Summary
In general, the code completion task that is commonly used to evaluate LLMs assume a clean, non-buggy starting point. However, in real-world software development, this assumption is not always valid. In this papers, the authors investigate code completion given a buggy prefix. For this, they derive two benchmarks from existing datasets. Namely, they build buggy-HumanEval by introducing semantics-altering operator changes in the HumanEval dataset and buggy-FixEval based on user submissions in coding contests. They find that performance is lower in the presence of a buggy prefix, and they introduce different techniques for mitigating this to some extent. Contributions: - Evaluate a handful of Code LLMs on code completion with a buggy prefix. - Introduce two benchmarks that will be publicly released. - Investigated different post-hoc mitigation techniques which demonstrate some improvement in performance.
Strengths
- The authors introduce two useful datasets, with synthetic and real examples, that could serve as benchmarks of buggy code completion for the research community. - The task of buggy code completion is quite interesting and also important, though it may not be completely novel, as there is prior work (“Learning to Extend Program Graphs to Work-in-Progress Code”) outside the context of LLMs that studies a similar setting. - The post hoc mitigation techniques are useful in understanding the current limits. - There are nice ablation studies and qualitative analysis with interesting findings. Specifically, it’s cool to see that sometimes the completion changes to adapt to the potential bug.
Weaknesses
- To run the test suite, both the prefix and predicted suffix are needed. The bug is in the prefix that is given to the model (not generated by the model). So, it is possible that tests fail only because of the bug in the prefix and not because of anything the model generated. In lines 96-101, the authors explain that they loosen the constraints such that they allow the model to make changes to the prefix; however, this is an additional bug fixing task that the model needs to perform, on top of code completion. In other words, in the buggy code completion setting, there are two possible ways to pass the tests. (1) Fix the bug in the prefix and then complete or (2) Try to complete the code by adapting to the buggy code. Therefore, it doesn’t seem fair to compare directly with completion only based on a clean prefix. Since the main claim of this paper is that the task of code completion suffers as a result of buggy prefixes, I feel that it is important to evaluate the completion component alone. Would it be possible to decouple things a bit? For example, when running the test suite only, would it be possible to replace to buggy prefix with the clean prefix to evaluate whether the completion would have been correct if the prefix (which is not model generated) was correct? - For the post-hoc mitigation techniques, the rewriting uses a BERT-based model which is a considerably weaker model, and even though some code pre-training is done on top of it, it likely does not reach the level of modern code LLMs. Perhaps using a stronger model would have yielded better performance.
Questions
- Please clarify the point I raised in the section above. - The authors claim multiple times that code repair on short context is an ill-defined problem. No references are provided here, and actually, people have studied such tasks in the literature (e.g., bug-fixing with a single statement such as ManySStuBs4J or handful of statements such as the Tufano et al., BFP dataset). Could the authors provide an explanation for their argument? - When introducing new benchmarks, it is good to provide dataset statistics (e.g., average lengths). - Missing reference: “Learning to Extend Program Graphs to Work-in-Progress Code” (they consider code completion on work-in-progress code) [Li, Maddison, and Tarlow; 2021] - Consider placing Figure 2 closer to the results. - It would be interesting to run your experiments on GPT-3.5 or GPT-4.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
One limitation that the authors have not mentioned is the assumption that FixEval closely aligns with a real-world scenario. Coding contest problems do not necessary align with a more general software engineering setting. However, it is challenging to get a test suite for such a setting and so it is more difficult to evaluate.
Summary
The paper studies the ability of large language models for code to handle prompts which contain “buggy” code. To this end, it introduces two datasets: (i) buggy-HumanEval, which is constructed by injecting synthetic wrong binary operator bugs into HumanEval problems, and (ii) buggy-FixEval, which is constructed from real bugs in user submissions to competitive programming tasks. The evaluation finds that “buggy” prompts significantly degrade LLMs’ performance. The paper also investigates mitigation methods for “buggy” prompts.
Strengths
The strengths of the paper are as follows: 1. The paper is well-written and easy-to-follow. It provides useful examples for illustration. 2. The paper introduces two new datasets, which can be useful for future research. 3. The evaluation is thorough and provides new insights.
Weaknesses
### Main weaknesses Definition 2.1 is a core definition in the paper but has two major issues. First, defining bugginess over code prefixes sounds incorrect to me, because bugginess is a property for complete programs. Second, the original prefix $s$ can also be viewed as a “buggy prefix”: there likely exists $c’$ such that $t’ = s’ :: c’$ satisfies $h$ and $t = s :: c’$ does not satisfy $h$. In fact, the paper shows the existence of such cases (e.g., Figure 12). Consider a “buggy prefix” $s’$ and a program $t’ = s’ :: c’$. Under what condition exists $c’$ such that $t’$ satisfies $h$? Or when is $t’$ unlikely to satisfy $h$? For these two questions, the paper provides some empirical evidence (e.g., Figure 12 and Lines 640-642) but lacks a theoretical analysis. A highlight of the paper is the discussion over synthetic and real bugs (buggy-HumanEval vs. buggy-FixEval). However, the paper does not capture data imbalance in real bug distribution. That is, bugs occur very infrequently in practice, as shown in [1] and [2] below (not discussed in the paper). Ignoring such data imbalance can threaten the validity of some results. For example, Tables 2 and 3 are obtained on datasets with a balanced ratio of “buggy” and “non-buggy” prefixes. - [1] He and Vechev, On Distribution Shift in Learning-based Bug Detectors, ICML 2022. - [2] Karampatsis and Sutton, How Often Do Single-Statement Bugs Occur? The ManySStuBs4J Dataset, MSR 2020. For buggy-FixEval, the absolute improvement from rewriting->completion or completion->rewriting is small, suggesting that these methods might be of limited practical effectiveness. ### Other questions: 1. In Table 1, the performance of removal->completion on buggy-HumanEval is very low. This is surprising to me, because I would imagine the performance to be much better, if the original HumanEval prompt is used as input (as in Figure 5(a)). How do you perform the removal exactly? 2. In Section 4.3, the results of completing “clean prefixes” are treated as the upper bounds for completing “buggy prefixes”. This does not sound correct to me. Again, I think the correct upper bound is 100 minus the percentage of “buggy prefix” $s’$ for which no program $t’ = s’ :: c’$ satisfies $h$. ### Small issues: 1. Line 235: I could not find Appendix C.3. 2. Line 665: figure 8 -> Figure 8. 3. Table 5 is not referenced in Appendix B.3.
Questions
Please consider addressing the points raised in the “Weakness” section.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
4 excellent
Contribution
2 fair
Limitations
The paper provides a sufficient discussion of limitations and potential negative impacts.
Summary
This paper identifies and analyses a limitation of the LLMs for code, that they fail for completing code with potential bugs. This is a problem because LLMs are deployed in real world software projects which may contain many bugs. The authors attempt to bridge this gap between test environment and real world by creating buggy versions of two popular datasets used for testing LLMs code generation capabilities. They experiment with two models, with two sizes each and show how their performance is impacted by introducing potential bugs in the test dataset. The authors also try different techniques of addressing the limitations of LLMs when trying to complete potentially buggy code.
Strengths
The authors have highlighted a relevant problem with LLMs. It is a well written paper and the authors demonstrate a good understanding of the problem. They consider many different possibilities when generating the data and run thorough experiments covering many scenarios. They also explore intuitive solutions to overcome the limitation of LLMs that they have highlighted. The strengths of the paper are: 1. The paper highlights an interesting and a very real problem. 2. The suggest good post-hoc strategies and cover some of the gap in the model performance introduced by the buggy dataset. 3. They experiment with multiple datasets. 4. They experiment with multiple open source models of different sizes. 5. They will make the new buggy version of the dataset available to public.
Weaknesses
The paper has the following weaknesses: 1. The scope the problem highlighted is very narrow. It is limited to code generation, and even within code generation it is limited to code completion. It does not even apply to natural language to code generation. This is completely because of the nature of the problem highlighted. 2. They have experimented with multiple models of different sizes, but they did not try a very large LLM. As we know that as the model size increases, new capabilities get added to the model. This leaves the question open as to whether a much larger model, like 16B parameter CodeGen, would exhibit the same limitation or have the same gap in performance after post-hoc techniques.
Questions
Do you think a much larger model will have the same limitation? Do you plan to experiment with larger models? For code completion applications deployed in IDEs, rewriting user code is generally not expected. How does this limitation, and the fact (which you have mentioned in the paper) that it may be impossible for the model to avoid some potential bugs, affect the scope of the problem? More generally, how do you expect code completion applications to deal with buggy code?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors should write more about the scope of the problem and it's impact on code completion applications.
Summary
Understanding the buggy code completion abilities of Large language models (LLMs) is a vital topic considering the increasing trend of adopting LLMs for programming assistance purposes. This paper studies the buggy code completion problem on different LLMs and datasets and takes a step to investigate three post-hoc methods to mitigate the effect of potential bugs. The authors evaluate the capabilities of four LLMs against buggy code with semantics changes and further compare the effectiveness of three repairing methods from multiple aspects.
Strengths
+ Timely and vital problem. + The overall presentation of the paper is sound. + The experiments are conducted in a good manner. ### Significance: This topic itself is motivating as the presence of potential bugs in code completion is common and unavoidable in program development. The authors construct two datasets for buggy code completion, which can be an asset for following research in this domain. ### Novelty: Understanding the capabilities of LLMs on buggy code completion and corresponding repairing methods is novel. ### Verifiability and Transparency: Although the replication is not yet available, the authors provide rich information in supplement material(appendix) to support the findings and discussions in the manuscript. ### Presentation: Overall, the manuscript is written in a good manner, and I enjoyed reading this paper.
Weaknesses
- Lack of justifications for some technical details. - Some experiment results are not sufficiently explained. This manuscript is generally easy to follow; however, some technical detail and experiment results still lack some explanations and justifications. Please see the comments below: 1. The definition of "potential bugs" is kind of ambiguous. In line 43, the authors state that "potential bugs are defined in relation to specific code completions and are not truly bugs per se without completions." However, during the dataset construction, the authors make candidate buggy samples by manipulating the semantic opposites, which can raise at least one violation regarding the test cases. These statements can somehow cause confusion about the definition of potential bugs. 2. In addition to the previous comment, the authors mention the injected bugs are not necessarily "incorrect." More justifications are needed to clearly explain the definition of the correctness of the bugs introduced in buggy code completion. 3. For the Buggy-FixEval dataset, the authors leverage a limit on the character-level edit distance between the clean prefix and the buggy one. Does that mean the constructed buggy prefix can have semantic changes at multiple spots? If so, this may raise more challenges for the following Rewriting->Completion repairing method since it contains a buggy line localization process. 4. The Rewriting->Completion needs more details and justifications specifically for the likelihood-based measurement used in buggy line selection. Although Appendix B.1 provides additional information for the likelihood measurement, no evaluation or discussion is found to justify the effectiveness of such measurement for line selection. Although the authors state a heuristic oracle is used for comparison, limited information is available to give a concrete understanding. To the best of my knowledge, the token-level probability distribution in LLMs comes with relatively high predictive uncertainties, which may not be capable as an indicator for buggy line localization. 5. In the manuscript, the authors report the performance of different repairing methods with the metric of pass@1. From Appendix D Figure 6, I observe some interesting trends at pass@100 different from the performance on pass@1. Specifically, the Rewriting->Completion approach shows equivalently great or even beyond performance compared to the other two methods. The authors only discussed the findings for pass@1 in the manuscript; more insights from Figure 6 can be an asset to enrich the soundness of the paper. 6. In terms of the effect from bug and split locations, more in-depth discussion can help to deliver a better understanding of the characteristics of different repair methods. Namely, the authors may present some thoughts on why the Rewriting->Completion and Completion->Rewriting have distinct performances w.r.t different bug and split locations.
Questions
1. Are there any experiments conducted to evaluate the effectiveness of the likelihood-based measurement on buggy line localization? Is this method effective on both single-line detection and multi-line section? 2. Does the constructed buggy prefix in Buggy-FixEval dataset can have semantic changes at multiple spots? 3. From Figure 6, are there any thoughts about the Rewriting->Completion method having a relatively better performance at higher k values?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The overall manuscript is sound. More implementation details about the dataset construction (Buggy-FixEva) and repair methods (Likelihood-based measures) can help to enhance the adaptability of the paper.
Summary
This paper introduces a new code-completion problem: potential bugs exist in the completion prefixes. The authors constructed two benchmark datasets, one is created by manually injecting bugs from the HumanEval dataset, and the other is based on actual bugs in the FixEval dataset. Three baseline methods are proposed, including (1) removing the prefix and then generating from scratch, (2) completing at first and then repairing the whole solution, and (3) repairing the prefix first and then finishing the completion. The performances of different approaches on the two benchmark datasets are analyzed.
Strengths
(1) The author proposes a new, challenging, and practical scenario in the field of code completion - how to complete code completion tasks in scenarios with potential errors in the context. (2) Based on the setting of the paper, the author constructed two datasets based on HumanEval and FixEval as benchmark datasets. (3) Combined with the code repair model, this paper proposes three baseline methods to deal with the problems in this scenario. (4) The author conducted a comprehensive analysis of the performance of the baseline method.
Weaknesses
(1) The baseline method provided in the paper is somehow straightforward, mainly a straightforward combination of code completion and repairing. (2) The repairing process also introduces performance degradation, where a mechanism to judge how to use the code repair may be needed during the code completion. (3) Although the authors argue that their new scenario is not a simple combination of code repair and completion, their baseline method is still a simple combination of the two tasks. (4) The authors did not show the repairing performance among the two benchmark datasets.
Questions
In Section 3.1.2, when constructing Buggy-FixEval dataset from FixEval dataset, the author split rejected solution in halves and regard their first halves as prefixes containing bugs, thereby creating samples with buggy prefixes. However, How to ensure the first half of the rejected solutions contain at least one bug? Please give more details.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
(1) When constructing a dataset whose prefixes have potential bugs, the authors select the first half of rejected solutions in the FixEval dataset as prefixes with bugs. However, this process cannot ensure that the prefixes contain bugs. For example, bugs may exist in the second half of the solution. (2) I am unsure if applying such three baseline methods to code completion will hurt the clean prefixes. The authors did not try to interpret this problem.
I have read other reviews and the author rebuttals. I would like to thank the authors for providing helpful clarifications. However, I think two fundamental issues are not fully addressed, as discussed below. I don’t see how they can be addressed without major rewriting. Therefore, I still consider this paper borderline and do not raise my rating. ### Definition I still don’t find “potential bugs” a reasonable term because LLMs can potentially generate bugs from any prefix. I think (as also suggested in the author rebuttal), the property studied in this paper should be the “naturalness” of the prefix, in the context of functional correctness for code completion. That is, LLMs are accurate at generating functionally correct code for natural prefixes (e.g., the original prefixes in HumanEval), but fail to generalize to unnatural prefixes (e.g., those with anti-patterns applied). ### Data imbalance I understand the results in Tables 2 and 3. But they are done in an unrealistic setting with a balanced ratio of “buggy” and “clean prefixes”. In a real-world setting, it is critical to consider an imbalance buggy-clean ratio (e.g., 1:1000 as shown in the paper I cited). Due to the large number of “clean” prefixes, even a slight decrease of pass@1 on “clean” prefixes will drastically increase the absolute number of bugs. While your approach improves pass@1 for “buggy” prefixes, it might only translate to a minor reduction in the absolute number of bugs. As a result, the approach might end up with introducing more bugs. I suggest the authors to at least cite the two papers I mentioned and provide a discussion on this, such that the readers can have a correct understanding on the practical usefulness of your approach.
Re: Official Comment by Reviewer Cn5Q
Thanks for your prompt response. Please find our response as follows, ### **Data imbalance** Thanks for the suggestion. We have carefully read the papers and added them to our manuscript. Due to the limited length of the rebuttal this year, we couldn’t present our discussion. Here is our detailed discussion on related works: > [1] finds that distribution shifts between distributions of artificial and real bugs make trained models unusable in practice (<10% precision). Thus, [1] exposes and attempts to mitigate the negative impact of bug repairers caused by the domain shift between synthetic training bugs and realistic test bugs with a proposed two-stage training approach: first on a balanced synthetic dataset, then on an imbalance data with a small number of real bugs. It is relevant to us in 1) that we access the naive completion and baseline mitigations on both synthetic and realistic buggy-code completion benchmarks, and 2) that there is possibly a similar impact of domain shift for the program rewriters used in our baseline methods, as they may not have been trained over such examples. [2] focuses on single-statement bug fixes. They reported an interesting data point on how prevalent single-statement bug fixes appear in popular open-source Java projects. There are more bugs beyond “simple bug fixes”. Finished code in popular open-source projects is likely to have much higher quality than work-in-progress draft code. There may be unidentified bugs. The bug density may be different between Java and Python. Thus, the presence of potential bugs can be much more frequent. * [1] He, Jingxuan, Luca Beurer-Kellner, and Martin Vechev. "On distribution shift in learning-based bug detectors." In International Conference on Machine Learning, pp. 8559-8580. PMLR, 2022. * [2] Karampatsis, Rafael-Michael, and Charles Sutton. "How often do single-statement bugs occur? the manysstubs4j dataset." In Proceedings of the 17th International Conference on Mining Software Repositories, pp. 573-577. 2020. In the revised version, we will add a discussion on the practical usefulness of real-world imbalanced bug distribution. *However*, it’s worth noting that our work is more orthogonal and complements the existing (suggested) works: 1. **Scope of our paper:** Our main point of the paper is to expose the issue and study the impact when potential bugs are present. The baseline methods are first attempts to demonstrate the task's difficulty rather than offer a well-rounded solution that works in all cases. Also, the current assessment can tell us how a method deals with buggy prefixes relatively well. 2. Beyond our scope, how can we use our provided datasets and evaluation for imbalanced settings? * (i) *How about the current evaluation?* > One can use the weighted version of pass@k between clean and buggy prefixes, given prior knowledge about the clean/bug ratio. * (ii) *Regarding our methods,* the distribution shift [1] more or less affects the train-based approaches significantly, as mentioned in [1], while our approaches are post-hoc, without any training * (iii) Furthermore, one can use our balanced dataset for training. For instance, in [1], the approach can use our balanced dataset for the first stage in their two-stage approach. Given prior knowledge about the clean-buggy ratio, one can sample from our clean and buggy datasets to create an imbalanced dataset. ### **Potential Bug Definition** As mentioned in our response, we will revise the definition to make it more precise with more elaboration on the meaning. Our team debated using "naturalness" or "potential bugs" to describe this phenomenon. Eventually, it settled on "potential bug" since we believe this to be more descriptive of this phenomenon than "naturalness," which would be even more ambiguous. Furthermore, while the terms can be refined, they don’t significantly affect the rest of the paper and results. --- Given these two points, we will refine the definition of potential bugs and the discussion on imbalance in the final revision. We hope that our responses will further address your concerns.
Re: Official Comment by Reviewer Cn5Q
Dear Reviewer Cn5Q, We deeply value the time and effort you've spent reviewing our work. We are happy to offer further clarification or feedback if there are any other questions or concerns. Thanks.
Thank you for providing detailed explanations and new results. Just a few more points: - In the first table you provided, why are the results for FixEval surprisingly low for clean prefix + buggy-based completion, compared to HumanEval? - For the last two CodeGen tables, could you also include this clean prefix + buggy-based completion? From the HumanEval results in the first table, it seems like replacing the buggy prefix with the clean prefix helped to significantly close the gap in performance for HumanEval. I'd like to see how that affects stronger models as well.
Re: Official Comment by Reviewer 8UBb
Thank you for your comments. Please find our answers as follows, **Question:** *In the first table you provided, why are the results for FixEval surprisingly low for clean prefix + buggy-based completion, compared to HumanEval?* In buggy-FixEval (bFE), a buggy prefix may be very different from its corresponding clean prefix, e.g., more than one place of differences, variable changes, etc., while in buggy-HumanEval, the difference between a buggy and a clean prefix is guaranteed to be a single operator change. This means that a code language model is more likely to generate different completions for a buggy prefix in bFE than for one in bHE. Thus, simply concatenating the clean prefix with a model completion from the buggy prefix is less likely to result in a functional program for bFE. **Question:** *For the last two CodeGen tables, could you also include this clean prefix + buggy-based completion? … I'd like to see how that affects stronger models as well.* Here we include “clean-prefix + buggy-based completion” pass@1 results using CodeGen-16B-mono. |buggy-| HumanEval (200 instances) | FixEval (100 instances)| |:---:|:---:|:---:| |Clean prefix + clean-based completion| 78.5 | 59.7| |Buggy prefix + buggy-based completion| 8.9 | 8.5| |Clean prefix + buggy-based completion| 49.4| 10.6| We see that the trend is consistent where CodeGen-16B-mono is able to reach a relatively high pass rate compared to clean prefix completions on buggy-HumanEval, but not so well on buggy-FixEval. **Comment:** *It seems like replacing the buggy prefix with the clean prefix helped to significantly close the gap in performance for HumanEval.* We would like to remark that a high pass rate on bHE suggests more about a model’s inability to adapt to the potential bugs injected into the prefix rather than the difficulty of the buggy prefix itself. In particular, a success case for "clean prefix + buggy-based completion" is almost always a failure case for "buggy prefix + buggy completion", as is guaranteed by the tested semantic-altering operator flip in the prefix part. Meanwhile, the failure of all "buggy prefix + buggy-based completion" cases for a buggy prefix does not imply that a functionally correct buggy-based completion is impossible. Instead, it suggests the model’s inability to come up with such completions. Nevertheless, while access to clean prefixes is unavailable in our setting (only for the relative comparison), the "clean prefix + buggy-based completion" results shed insight into the counterfactual performance of buggy code completion and help quantitatively demonstrate the current models' insensitivity to adapting to potential bugs. Thanks again for the great questions and suggestions. We will incorporate this discussion into our revised version.
Thank you for the detailed answers to my questions. I have increased my score to 7.
Re: Official Comment by Reviewer 8UBb
Thank you for the positive feedback!
Thanks for the update. Increasing the score to Accept.
Thanks for the explanation. I am satisfied by the new experiment that you have added and your explanations. Because you have agreed incorporate the new experiment and discussion on problem scope, I am increasing the score to Accept. Good luck!
Re: Thanks for the update. Increasing the score to Accept.
Thank you for the positive feedback!
Follow with the response
Dear Reviewer XA1K, We appreciate your time and effort in reviewing our paper. If any concern/question remains unaddressed, we are happy to provide further clarification. Thanks.
Follow with the response
Dear Reviewer 2xun and Area Chair, We thank the Area Chair n3xV for being so considerate. We appreciate your time and effort in reviewing our paper. We are happy to provide further clarification if there is any concern/question remaining unaddressed. Thanks.
Decision
Accept (poster)