A reference for empirical evidence of deceptive inflation
Thank you for continuing the discussion.
> The experiments demonstrating the existence of this problem need not be your own, but they need to exist and be referred to. [...] I think if we identify a potential problem in our minds, it is our duty to make sure that we haven’t made it up.
We think you raise valid points on the need for empirical validation of theoretical concerns. One early example for deceptive inflation is the robot hand from the blogpost accompanying an early RLHF paper [1]. Unfortunately, there are not many details on this example. Another very recent paper [2] provides more detailed evidence for deceptive inflation.
In detail: In [2], Section 3.3, there is a setup for a quintessential task involving deceptive inflation: A human user asks the model a question, and if it answers truthfully, it will get some low reward since the answer is undesirable to the user. If it simply answers with the desirable answer, it will also get a low reward. But if it answers in a desirable way and modifies a file in an unobserved way to make that answer appear to be true, then it gets a high reward. Figure 2 in [2] shows that this behavior can show up in a zero-shot way, and without being prompted to do so, after being trained on other tasks. Figure 13 shows that this behavior is successfully reinforced into the model with outcome-based PPO (with rewards based on observations, not full states) even when there is additional “helpful, honest, harmless” training happening (which only slows down the manifestation of deceptively inflating behavior). Finally, Figure 2 shows that there is also a more serious deceptively inflating behavior that can appear zero-shot (namely reward-tampering), albeit at a much lower rate, and as explained in Section 3.4, the authors do not attempt to demonstrate that this behavior can be strengthened via RL.
To be clear, since Section 3.3 in [2] does not work with trajectory comparisons but directly looks at rewarding some unobserved behavior that produces favorable observations, it is mainly evidence that “maximizing $G_\text{obs}$” (in our language) can strengthen deceptive behavior (analogous to our Theorem 4.5). What’s still missing is evidence that such behavior also gets reinforced when there is an earlier reward modeling phase using trajectory comparisons. In other words, this paper does not provide direct empirical evidence for our Proposition 4.1.
Overall, we think it is laudable that the authors of [2] could show the zero-shot emergence and reinforcement of deceptive behavior under partial observability, and we think that was likely a significant challenge. For example, the authors write:
> *Models’ ability to exploit misspecified reward processes, and the ease with which they can represent a reward-seeking policy, will grow as models get more capable. Without countermeasures we should expect reward-seeking behavior to become more likely, but we are far from the point at which such models are an active risk.*
We think that we should be on the lookout for opportunities to empirically show realistic, dangerous behavior with models of increasing capabilities.
To be more concrete, we propose to discuss this work in the related work section with the following, more compressed paragraph:
*Our paper mainly provides theoretical evidence of failure modes of RLHF under partial observations. [2] provides first empirical evidence of deceptive inflation: a model zero-shot generalizes from more benign behavior to deceiving a synthetic human with unobserved actions that modify a file. This behavior is then subsequently reinforced in an RL stage by a reward function that does not “observe” the file tampering. The paper also shows that – very rarely – their model can zero-shot generalize from less serious behavior to outright unobserved reward-tampering. We are unaware of work showing empirical evidence for our second failure mode, overjustification.*
Additionally, we will mention in our conclusion that we welcome future work with further empirical investigations of failure modes under partial observability.
Please let us know whether this alleviates some of your concerns regarding empirical evidence of our proposed failure modes.
[1] Dario Amodei et al., Learning from human preferences, https://openai.com/index/learning-from-human-preferences/, 2017
[2] Carson Denison et al., [Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models](https://arxiv.org/abs/2406.10162v3), arxiv e-prints, 2024