We thank you again for taking your time to read our responses and provide suggestions.
> **1. The analysis of token cost appears to assume an ideal scenario where no failures occur.**
Indeed, in the analysis of general response, we simplified the situations and ignored the the situations where the LLMs make a mistake. However, in token cost results (Table 3, 11 and in our response) we have **always included** token usage for retries to ensure fair comparison, and we argue that this situation may be trivial to consider.
Failures that cause retries can rarely happen in FunCoder -- they happen if and only if no code can be extracted from the responses' Markdown blocks, or that the code was syntactically incorrect. We provide additional statistic results in the table below, which shows a relatively low failure rate even in complex questions (MATH or xCodeEval).
| Dataset | Pass@1 | failed to parse / all LLM calls |
|---|---|---|
| HumanEval | 85.4 | 0.10% |
| MBPP | 78.5 | 0.31% |
| xCodeEval | 31.4 | 1.68% |
| MATH | 54.0 | 2.10% |
> **2. We would like to see the analysis results of Table 1 or 2 (compared to the SOTA).**
We report token consumption data for some of the fields in Table 1 and Table 4, regarding GPT-3.5 and SOTA methods, in the table below. Some of the cells are missing data since some methods were not open-source and did not report detailed token usage.
| Dataset | Method | Pass@1 | min tks | max tks | mean tks | med tks |
|---|---|---|---|---|---|---|
| HumanEval | Standard | 68.3 | 648 | 1477 | 886.7 | 861 |
| | CodeT | 81.1 | 2298 | 9645 | 4479.1 | 4166 |
| | Reflexion | 69.5 | 416 | 4906 | 1416.1 | 754 |
| | LDB (Reported prev SOTA) | 82.9 | - | - | ~23000 | - |
| | FUNCODER | 85.4 | 3015 | 13850 | 5402.0 | 5166 |
| xCodeEval | Standard | 20.2 | 1051 | 3343 | 1599.5 | 1530 |
| | CodeT (prev SOTA) | 23.2 | 2264 | 9245 | 3937.4 | 3733 |
| | Reflexion | 20.6 | 2977 | 1003222 | 401767.3 | 328591 |
| | FUNCODER | 31.4 | 4883 | 53225 | 10559.7 | 8927 |
| MATH | Standard | 62.2 | 551 | 5867 | 953.0 | 835 |
| | FUNCODER | 76.8 | 2622 | 30139 | 7075.5 | 5666.5 |
> **3. Then, your success rate to derive b->B and c->C should be increased to the square root of 0.945, which means 0.972. When the program becomes much more complex, the requirement of success rate for each divided subprogram is further increased.**
**Accumulated errors in FunCoder also occur in Standard**, where generating all functions (ABC) all at once would have a higher error rate than generating one at a time (A, B, C).
Note that this metaphor may not be exact since 94.5% acc is an average of all problems, where different problems are decomposed into different depths. However, we follow this metaphor and continue to show why FunCoder works better than other methods. Here success rate of *Standard* on a single function is $\sqrt{82.9}=91.0$. On a harder dataset where problems are decomposed to 10 levels of functions on average, *Standard*'s overall accuracy will be $91.0^{10}=38.9$, while FunCoder still keeps $97.2^{10}=79.3$.
Results on xCodeEval dataset shows that FunCoder gets greater improvements on harder problems compared to simple problems. But of course, the analysis we just made was based on a simple assumption and does not reflect reality.
> **4. I think there should be some design and cost to increase the performance, even if the divided programs are simpler than the original ones.**
We designed FunCoder under the consideration of complex problem performance, and have made visible progress. Particularly, our method:
1. **Divide** dynamically decomposes problems into simpler subproblems, making them easier to solve. The Divide stage always considers just one layer of problem at a time, thus can keep the model in context, reducing ambiguity and complexity caused by extra-long code. For instance, the problem A-B-C-D requires all these functions to be in context in *Standard*, while ours only need 2 for each of the 4 calls.
2. **Conquer** enables bottom-up generation through re-composing simple functions into a complex solution to a complex problem. This mitigates the issue where Divide cannot see the subfunctions while generating parent functions, making function dependencies more robust. (Note that LLMs are autoregressive models, so while *Standard* generates ABC, when A is being generated B,C aren't there yet. This could cause *Standard* to mis-invoke B or C from A in this implementation.)
3. **Functional consensus** verifies every function starting from the leaves. Through sampling and voting, incidental errors and cascading errors may be reduced and thus the accuracy could be improved. Before our method, self-test was widely used for verifying programs, and now FunCoder manages to achieve higher performance with the same level of token complexity.
We further note that Divide, Conquer and Consensus may be used together, and that using them in conjunction will further raise the accuracy on complex problems.