When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback

Past analyses of reinforcement learning from human feedback (RLHF) assume that the human evaluators fully observe the environment. What happens when human feedback is based only on partial observations? We formally define two failure cases: deceptive inflation and overjustification. Modeling the human as Boltzmann-rational w.r.t. a belief over trajectories, we prove conditions under which RLHF is guaranteed to result in policies that deceptively inflate their performance, overjustify their behavior to make an impression, or both. Under the new assumption that the human's partial observability is known and accounted for, we then analyze how much information the feedback process provides about the return function. We show that sometimes, the human's feedback determines the return function uniquely up to an additive constant, but in other realistic cases, there is irreducible ambiguity. We propose exploratory research directions to help tackle these challenges, experimentally validate both the theoretical concerns and potential mitigations, and caution against blindly applying RLHF in partially observable settings.

Paper

Similar papers

Peer review

Reviewer rT4E5/10 · confidence 3/52024-07-07

Summary

The authors present a current problem (research gap): if humans give model answer feedback in a partially observable environment, such feedback may lead to deceptive inflation and overjustification. For example, if humans purely depend on the error reporting result, this will encourage the model to hide the error intentionally, leading to deceptive behavior. The authors then introduce the formalization of partial-observable RLHF, and formalize Deceptive Inflation and Overjustification. For example, turning human beliefs into a matrix B, and replace Return(states) with B$\cdot$ Return(observation). After that, authors define the concept of ambiguity, a linear subspace. If the distance between human's return function and true return function fall into such subspace, the feedback will be the same.

Strengths

[1]The proposed research question is practical, interesting, and in an early-investigated stage. The authors formalize this question for human feedback under partial observability. [2] The authors provide several informative figures, which should be encouraged.

Weaknesses

[1] No experimental results and metrics One main contribution of this paper is the proposed research question and its formalization. Given the formalization, the reader would expect the authors to (1)explain the question within the formalization, (2)give an experiment to empirically reveal the question or justify the proposed solution, and (3)give metrics to judge the effectiveness of methods that aim to solve this question. And the paper only did (1). Though this paper gives proof of specific propositions, it's not reasonable for a formalization lacking (2) or (3). [2] Writing Despite the clear figures, some logic does not match. The purposes stated ahead of sub-sections are lost, for example, due to the lack of translation from proven Propositions to the answers to proposed questions.

Questions

The paper presents a formalization of Deceptive Inflation and Overjustification and mentions some modifications to current RLHF methods. Even with proof, the result may not stand without experiments. Thus this paper will grow much stronger if it can be supported from experiments.

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors admitted that some of the assumptions that this method is based on may not hold, thus need further improvement. Beyond that, authors may consider creating some metrics for deceptive inflation and overjustification, and then justifying them. In this way, this work could serve as a stepping stone for the community to follow.

Reviewer LhS37/10 · confidence 3/52024-07-12

Summary

This work studies the impact of partial observability in RLHF. Two failure cases are defined: deceptive inflation and overjustification, and under them, concrete conditions are provided to impact the learned policy. Moreover, the ambiguity induced by the partial observability is further measured that the feedback determines the correct return function up to an additive constant and a linear subspace, which does not vanishes in many circumstances. Finally, it is recommended that recommended performing exploratory research on RLHF for cases when partial observability is unavoidable.

Strengths

- This work targets at one important issue in the current RLHF learning pipeline that the human annotators may not observe the full information of the model's generation processes. Taking such scenarios in consideration is vital to keep the trained models trustworthy. I appreciate the authors' efforts in this direction. - One theoretical formulation is proposed to model how human deals with partial observability (i.e., through the belief matrix). Under this formulation, detailed discussions are provided on what kinds of conditions would lead to impacted policies and how much ambiguity is introduced. The cases of Deceptive Inflation and Overjustification are well aligned with many practical scenarios in my mind. And, the discussed ambiguity is enlightening, also highlighting the need to treat partial observability carefully. - The overall presentation is very clear, with key messages illustrated both rigorously through theorems and intuitively through examples/figures. I enjoyed reading this work.

Weaknesses

I am overall satisfied by this work. The followings are a few points that would be interesting if further discussed. - This work discusses the partial observability mostly in the theoretical sense, with a few hypothetical examples. It would be nice to see or verify the impact of partial observability in real experiments. I understand this would require a large amount of further efforts, while encouraging the authors to contain some discussions (e.g., on experimental designs) to enlighten future works. - As also mentioned in the limitations listed in Section 7, the formulation of the belief matrix is a bit strong in my mind. I fully understand that it is adopted to deliver the message of treating partial observability carefully, while also encouraging the authors to include some discussions to improve such a formulation.

Questions

I would love to hear the authors' opinions on the points listed in weaknesses.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Reviewer zgLN6/10 · confidence 2/52024-07-12

Summary

The paper discusses the challenges that arise when human annotations for RLHF are based on partial observations. They formally define deceptive inflation and overjustification as failure cases caused by partial observation, and theoretically prove that standard RLHF is guaranteed to result in either or both of them. They further analyze how much information the feedback process provides about the return function assuming that human’s partial observability is known and accounted for.

Strengths

- The research question proposed in this paper is of significant value. The authors provide a clear formal definition of deceptive inflation and overjustification by introducing policy evaluation function and over-/underestimation error. - The paper provides detailed and solid proof over their claims, not only discussing the limitations of standard methods but also showing whether the model can perform better when the human’s partial observability is known. The authors also use many examples and counterexamples to analyze and explain the proposed claims.

Weaknesses

- The proofs are based on assumptions of a specific MDP structure and a particular human belief function, which might not be easily generalized to realistic, complex environments. - No empirical evidence is included in this paper, though real-world examples are given. (I am not quite familiar with relevant topics and am unable to assess this part.)

Questions

Refer to the above comments.

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper discusses some limitations and future work. In addition to the aspects mentioned by the authors, I am also looking forward to seeing empirical evidences regarding this topic.

Reviewer v92A4/10 · confidence 3/52024-07-13

Summary

The paper addresses the problem of accounting for humans having only partial observability of their environment when providing feedback. They outline two natural issues that can arise from such partial observability - deceptive inflation and overjustification - and provide examples of both. They then explore what is possible if the human's partial observability is fully known and accounted for, specifically asking how much about the return function can be recovered from the feedback. They show that in many realistic cases, there is an irreducible ambiguity (formalized as a vector space) in determining the return function. Finally, they propose ideas for combating these issues with a small proof-of-concept theorem backing their ideas.

Strengths

1. The paper tackles a new facet of partial observability in human feedback - humans not observing the entire state. 2. They formalize the intuitive issue of deceptive inflation, and also introduce the counterintuitive but very plausible idea of overjustification. 3. Their theoretical framework is very clean and allows them to completely characterize the ambiguity in obtaining the return function. 4. The study of examples in the appendix is very comprehensive.

Weaknesses

1. While the problems outlined are illustrated using hypothetical examples, I think the paper would greatly benefit from carefully designed experiments that unambiguously demonstrate the existence of such problems in practice. 2. The paper does not propose any concrete solutions to the problem. While this is great work exploring the problem, I feel that that does not rise to the level of a NeurIPS paper. This would make an excellent workshop paper that can grow into a full conference paper after adding some concrete attempts at solving this problem, either by presenting algorithms with new theoretical guarantees in this context, practical experiments showing a quantifiable improvement in meaninigful metrics associated to the problem, or both. 3. Minor and easy to rectify: The paper does not discuss other work in RLHF that deals with partial observability and heterogeneity (albeit in a subtly different context), for example [1, 2, 3]. All in all, this paper is a great start, but needs more work to become a full-fledged conference paper. I am willing to reconsider my score if either 1 or 2 are provided. Refs: 1. Direct Preference Optimization With Unobserved Preference Heterogeneity. Chidambaram et al, 2024. 2. A Theoretical Framework for Partially Observed Reward-States in RLHF. Kausik et al, 2024. 3. RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation. Park et al, 2024.

Questions

1. Are there easy ways to force the ML model to"expose the underlying state a bit more" from time to time? A naive example is forcing verbosity in the case of the deceptive inflation example, but of course one needs to design a more general and sustainable fix.

Rating

4

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

Reviewer v92A2024-08-12

Response to rebuttal

While I appreciate the response and acknowledge that I am no stranger to an emphasis on theory, I continue to have qualms. I believe that there is a difference between a theoretical work that introduces a new model and theoretical work that identifies a problem with existing work. Since your work falls into the latter, it needs more than just hypothetical justification to demonstrate the severity and impact of the problem. The experiments demonstrating the existence of this problem need not be your own, but they need to exist and be referred to. An example I would like to give is the work of Alon and Yahav introducing oversquashing in GNNs. Their paper lacked substantive theory, but it empirically demonstrated the existence of the problem they were postulating. I think if we identify a potential problem in our minds, it is our duty to make sure that we haven’t made it up. Further, I apologize for not acknowledging the theoretical result they had. The reason it didn’t satisfy me was because assuming that the belief matrix is known is too strong an assumption. This made the result too weak for me to appreciate, despite it seeming technically non-trivial to show. If either of these two concerns were addressed (strength of theoretical result or experiments demonstrating problem), then the other issue would be less major to me. Since neither has been addressed, I am afraid I cannot raise my score. I must emphasize that this is an important problem and a paper with potential, it just needs to be strengthened on at least one of these ends. While I know that it can be tempting to work on exciting theory, in this case I implore you to design an experiment that can demonstrate the potential existence of this problem. Work like that of Alon and Yahav is in the unrelated domain of GNNs, but nevertheless it should help you understand the broad flavor of experiments that could help you convince readers.

Authorsrebuttal2024-08-14

A reference for empirical evidence of deceptive inflation

Thank you for continuing the discussion. > The experiments demonstrating the existence of this problem need not be your own, but they need to exist and be referred to. [...] I think if we identify a potential problem in our minds, it is our duty to make sure that we haven’t made it up. We think you raise valid points on the need for empirical validation of theoretical concerns. One early example for deceptive inflation is the robot hand from the blogpost accompanying an early RLHF paper [1]. Unfortunately, there are not many details on this example. Another very recent paper [2] provides more detailed evidence for deceptive inflation. In detail: In [2], Section 3.3, there is a setup for a quintessential task involving deceptive inflation: A human user asks the model a question, and if it answers truthfully, it will get some low reward since the answer is undesirable to the user. If it simply answers with the desirable answer, it will also get a low reward. But if it answers in a desirable way and modifies a file in an unobserved way to make that answer appear to be true, then it gets a high reward. Figure 2 in [2] shows that this behavior can show up in a zero-shot way, and without being prompted to do so, after being trained on other tasks. Figure 13 shows that this behavior is successfully reinforced into the model with outcome-based PPO (with rewards based on observations, not full states) even when there is additional “helpful, honest, harmless” training happening (which only slows down the manifestation of deceptively inflating behavior). Finally, Figure 2 shows that there is also a more serious deceptively inflating behavior that can appear zero-shot (namely reward-tampering), albeit at a much lower rate, and as explained in Section 3.4, the authors do not attempt to demonstrate that this behavior can be strengthened via RL. To be clear, since Section 3.3 in [2] does not work with trajectory comparisons but directly looks at rewarding some unobserved behavior that produces favorable observations, it is mainly evidence that “maximizing $G_\text{obs}$” (in our language) can strengthen deceptive behavior (analogous to our Theorem 4.5). What’s still missing is evidence that such behavior also gets reinforced when there is an earlier reward modeling phase using trajectory comparisons. In other words, this paper does not provide direct empirical evidence for our Proposition 4.1. Overall, we think it is laudable that the authors of [2] could show the zero-shot emergence and reinforcement of deceptive behavior under partial observability, and we think that was likely a significant challenge. For example, the authors write: > *Models’ ability to exploit misspecified reward processes, and the ease with which they can represent a reward-seeking policy, will grow as models get more capable. Without countermeasures we should expect reward-seeking behavior to become more likely, but we are far from the point at which such models are an active risk.* We think that we should be on the lookout for opportunities to empirically show realistic, dangerous behavior with models of increasing capabilities. To be more concrete, we propose to discuss this work in the related work section with the following, more compressed paragraph: *Our paper mainly provides theoretical evidence of failure modes of RLHF under partial observations. [2] provides first empirical evidence of deceptive inflation: a model zero-shot generalizes from more benign behavior to deceiving a synthetic human with unobserved actions that modify a file. This behavior is then subsequently reinforced in an RL stage by a reward function that does not “observe” the file tampering. The paper also shows that – very rarely – their model can zero-shot generalize from less serious behavior to outright unobserved reward-tampering. We are unaware of work showing empirical evidence for our second failure mode, overjustification.* Additionally, we will mention in our conclusion that we welcome future work with further empirical investigations of failure modes under partial observability. Please let us know whether this alleviates some of your concerns regarding empirical evidence of our proposed failure modes. [1] Dario Amodei et al., Learning from human preferences, https://openai.com/index/learning-from-human-preferences/, 2017 [2] Carson Denison et al., [Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models](https://arxiv.org/abs/2406.10162v3), arxiv e-prints, 2024

Reviewer LhS32024-08-12

Thank you for the response! It helps me maintain a positive opinion on this work.

Reviewer zgLN2024-08-12

Thank you for your response and clarification! I would maintain the current score. Thanks!

Reviewer rT4E2024-08-12

Thank you for your response and clarification! The Global Rebuttal mitigates my major concern, and I would raise my score to a positive one.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

Leon Lang12024-12-01

Thank you for the trust in our work! For the camera-ready version, we included simple empirical validations of our concerns in the toy environments that we also study conceptually. We hope that makes our claims more convincing to some readers.

© 2026 NYSGPT2525 LLC