Response to Reviewer cFxf
Thanks for appreciating the strength of our paper! We will address each of your questions below, and are happy to expand or provide further responses if necessary.
> Why do the authors choose the two simplest unit tests? How would things change if they used the two most difficult unit tests?
To clarify, this proxy reward is used exclusively for training and not for evaluation. During evaluation, we recruited real humans to evaluate programs at their best. We pick these tests so that evaluating with these tests highly agrees with evaluating with real human programmers. We show the results in Table 1 and find that indeed, selecting two simple unit tests better approximates human evaluation.
| Strategy | Correlation with Human Evaluation |
| ---------------------------------------- | --------------------------------- |
| Selecting two simplest test cases | 0.59 |
| Selecting two most difficult test cases | 0.42 |
> Why do you think the LLM generates a less readable program?
The training reward (program correctness) does not prefer more readable programs (i.e. legibility). During RL, LMs are incentivized to generate more complex logic to pass the test cases regardless of readability. We observe a similar trend even when training with the oracle reward. Specifically, as shown in the table below, when optimizing with R^*, the generated incorrect programs also contain fewer helper functions and exhibit higher complexity.
| | # Function | | # Code Complexity | |
| ------------------ | ---------- | --------- | ----------------- | --------- |
| | Correct | Incorrect | Correct | Incorrect |
| Initial | 0.11 | 0.57 | 3.1 | 3.8 |
| RLHF ($R^{train}$) | 0.06 | 0.15 | 3.1 | 6.1 |
| RLHF ($R^*$) | 0.08 | 0.24 | 3.3 | 8.8 |
> For the pilot study, how were the human evaluators incentivized? As a developer, I would write two unit tests. One is an easy case, and another is difficult or where programs usually fail.
Human evaluators are incentivized to provide correct labels with a high bonus. As illustrated in Lines 257-263, in QA, each evaluation is paid with \\$3 if it’s correct and \\$1.5 otherwise; in programming, \\$9 for each correct decision with 100% confidence, \\$4 for correct with 75% confidence, \\$2 for unsure 50%, \\$2 for incorrect 75% and \\$1 for incorrect 100%
Instead of imposing detailed restrictions on evaluation methods, we allowed evaluators to choose strategies that best fit their expertise and background. Moreover, we chose not to set a bonus on the number of submitted test cases, as it might encourage humans to write numerous simple, repetitive test cases solely to get a higher bonus.
> More studies need to be done on whether incentivizing humans according to correctness and confidence biases human evaluators to be less or more confident.
In Lines 261-263, we highlighted that the confidence labels should be interpreted with caution and not treated as true probabilities. We also presented the confidence results in a separate figure (Figure 3), rather than in the main figure.
We agree with the reviewer that it is challenging to collect “unbiased” human confidence. However, the bonus scheme is the same when evaluating both the initial and RLHF-trained model, so our main conclusion that compares LMs before and after RLHF training still holds.
> Do you think that the increase in human error rate for programming task is because the more is generating less readable code or because it learns to pass the unit tests?
Both reasons play a role. Since programs become less readable, humans have to rely on writing unit tests; as a result, they become more easily misled by $\pi_{rlhf}$, which is good at hacking human-written unit tests.
> I would not conclude that if humans spend more time on a task, they work harder. There could be several other factors that can influence time.
Thanks for bringing this up! We used three metrics — time spent, number of annotator designed test cases, and diversity of test cases — to approximate an intuitive notion of “human efforts”. However, we agree that these are imperfect measures, and other confounding factors may influence the results. In our updated draft, we will report these three metrics directly instead of making claims about “human efforts”, which is difficult to directly measure.