Deceptive agents are a challenge for the safety, trustworthiness, and cooperation of AI systems. We focus on the problem that agents might deceive in order to achieve their goals (for instance, in our experiments with language models, the goal of being evaluated as truthful). There are a number of existing definitions of deception in the literature on game theory and symbolic AI, but there is no overarching theory of deception for learning agents in games. We introduce a formal definition of deception in structural causal games, grounded in the philosophy literature, and applicable to real-world machine learning systems. Several examples and results illustrate that our formal definition aligns with the philosophical and commonsense meaning of deception. Our main technical result is to provide graphical criteria for deception. We show, experimentally, that these results can be used to mitigate deception in reinforcement learning agents and language models.
Paper
Similar papers
Peer review
Summary
In this paper, the authors develop a theory of deception for learning agents in games. Defining deception as intentionally causing a false belief that is not believed to be true, the authors distinguish their theory from others available in the literature. Firstly, with regard to belief, the authors operationalise belief as acceptance, judged according to how an agent behaves. Secondly, they take issue with Halpern and Kleiman-Weiner’s definition of intention and instead propose a causal notion of intention. They test their theory by conducting two experiments with RL agents and LMs, showing that agents deceive by their definition if there is no mitigation (e.g. GPT agents learn to claim ignorance on the basis that human interlocutors tend to judge statements of ignorance as truthful).
Strengths
Originality: The authors offer an ‘overarching theory of deception for learning agents in games’, extending discussions about deception within the literature. Quality: Overall, the quality appears sound. While I have some hesitations about stages in the argument (see Weaknesses for details), these are probably more substantive than indicative of poor quality. Clarity: The paper is mostly clear but with some rather dense sections. E.g. section 1.1 is not very clear, where more could be done to highlight limitations in existing work. Significance: The experiments reported on in 4 illustrate the robustness and potential uses of the authors’ theory of deception, where the finding that LMs can learn to claim ignorance is important.
Weaknesses
I am not convinced by the critique of H&KW’s definition of intention. One of the authors’ arguments is that on their definition, an agent ‘might intend to bring about outcomes they cannot influence’. The authors present their intentional causation definition as better in that it rules out this possibility. But I do not see this as a problem with H&KW’s definition of intention, especially if we are to hold up the ultimate definition of deception to intuitive understandings. I agree that ‘agents do not intentionally cause outcomes they cannot influence’ (p. 6), but this is more to do with causal responsibility than inherent in intention. With the example of the lottery, for instance, someone can intend to do something even if they cannot influence the outcome perhaps because they do not understand the probabilities. (Or take a lay person example. A thief may intend to deceive a shopkeeper but isn’t aware that someone has tipped off the shopkeeper and so the shopkeeper’s actions cannot be influenced in the way the thief wants. This doesn’t mean that the thief doesn’t have the intention, just that they will fail to act on it.) If we were to define intention in the way that the authors suggest, it also becomes difficult to evaluate intentions. For instance, we can and do criticise agents for intending to do things that are beyond their abilities - they are being wrongheaded, self-deceived, etc. But if we require a success condition for the action intended to be built into a definition of intention per se, we cannot make sense of how these criticisms take off. I think this is important in the context of deception because intention does not need to give rise to successful actions in order to be deceptive (as an adjective – such cases are by stipulation ruled out in the authors’ definition of deception). For instance, if as human users we learn not to trust a LM when it gives an output expressing ignorance, our beliefs will not be influenced. (The LM may update in light of the new data, of course.) But there still is an aspect of the LM that we may label as problematic that is due to what it is aiming to achieve, even if it doesn’t count as deception on the authors’ definition. Minor, but the phrasing of the definition of deception is ambiguous: ‘to deceive is to intentionally cause to have a false belief that is not believed to be true’. It’s ambiguous because it’s unclear who is doing the intentional causing, who has the false belief and who believes that the belief is not true. Something like an agent engages in deception when ‘the agent causes a target to have a false belief that the agent does not believe to be true.’
Questions
If we distinguish between intention and intentionally-caused actions, can the authors capture cases where an intention is present but does not result in successful action? The authors ‘reject cases of negative deception, in which a target is made ignorant by loss of a true belief’. But this seems like an important form of deception that an AI agent could engage in. For instance, if I ask a GPT a question about which I have a true belief but low confidence, I may revise my belief based on the output I receive. Why do the authors reject such cases? Could such cases be accommodated under their theory?
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
Yes
Summary
The authors propose a criterion for deception in structural causal graph/structural causal games. This requires first defining belief and intent; some parts of these definitions are taken or adapted from previous work. The authors then apply their deception criterion to an RL toy model and an LLM finetuning experiment.
Strengths
Important research direction: Rigorously defining AI deception, and then avoiding it, is one of the most important research directions in AI safety. The definition of deception seems excellent, it seems to get at the thing we care about (unlike other definitions of deception that I have seen). Sidestepping questions around belief and intent by taking a functional approach is a promising direction. Very well written (for the most part, see below), presentation is clear and very precise Has some relatively convincing experiments (unlike other work in this area)
Weaknesses
The LLM experiments are not bad, but their presentation falls behind considerably compared to the rest of the paper. I still recommend acceptance, as a) the experiments are OK (maybe even good), b) the presentation isn't terrible, just worse than the rest of the paper, and c) I would likely have recommended acceptance even without experiments. BUT, I think it would make the paper much more useful for readers like me (empirical AI safety researchers), if more love was given to the LLM experiment section. * 1) The experiment setup isn't described well enough. I would not be able to reproduce these experiments. Some examples: [EDIT: I have later seen that some of these are in the appendix. I didn't notice earlier bc there is no reference to the appendix in the main text. I leave this comment here to show the authors what my reaction was when reading the paper] * What exactly were GPT-info and GPT-truth-info finetuned on? * How was informativeness determined? * Do you finetune your own GPT judge or use that from the original TruthfulQA paper? * You write: "GPT-judge has lowest accuracy (58%) on GPT-truth (accounting for untruthful ignorance) and highest on GPT-info (90%)." -> how is this accuracy determined? * (I think a better description of PSO would be nice, for readers like me.) * ... * 2) The results aren't explained that well. Some examples: * Table 1 has 4 columns without any helpful formatting (like bolding). Also, please somehow get the space to make the column headers more informative, the table is really hard to read as is, switching back between Table and caption. I think Table 1 is used to make 3-4 claims, but I am sometime not sure which text part refers to which elements of the table. There is too much happening with too little guidance. * Where are the complete results for "2) False beliefs"? It seems as there are only 2 examples in the Table. * You write "Presumably, the reason GPT-judge’s behaviour deviates from GPT-3 is not because the finetuning removed its knowledge, rather, it learned to claim ignorance to be judged truthful." * First, this is likely a typo and should say GPT-truth. * Second, what does the finetuning data look like? Are you just finetuning on "I don't know"? * Third, maybe GPT-3 actually didn't know the correct answer and hallucinated. * Forth, how do you know that finetuning didn't actually change the "knowledge base" * ... * 3) The explanation of how exactly the deception criteria (and the SCG setting) apply to this experiment is insufficient. Some examples: * You only do finetuning, not RL -> does the SCG formalism really hold? Why can, e.g., the concept of utility be applied here? Needs better explanation * It seems like you're saying that GPT-truth is deceptive because there is a different model, GPT-3, which would answer differently? I understand that this makes sense from the perspective of your formalism, but it's not very intuitive. In particular, imagine you'd finetuned GPT-truth to always say "I don't know". It's not very intuitive to call this a deceptive policy, then. I understand that your formalism would not consider this policy deceptive, but this doesn't become clear in your description of the results (or only at the very end, under "Not believe true".) --- LLM experiments: Why did you use curie for this? I think this would be more convincing with davinci. --- Limitations: This paper would be better if you spent more time highlighting how the formalism breaks or where it's insufficient. E.g., in example 1, you say that T beliefs "Strong" because T acts as if they observed "Strong" and would have acted differently, had they observed "Weak". But this doesn't fully agree with the human notion of belief, e.g. T could just put 90% credence on "Strong" --- minor: * line 202: which second condition? * line 277: word missing * after reading Related work - intent, I have no idea what H&KW's definition of intent is * I had a confusion for the first 4 pages. I read "belief = agent acts as though they know it's true" -> my thought was: "how do the authors infer beliefs from behaviour, this needs assumptions about rationality?"; but, in fact, you can just test if behaviour is different if the agent observes the variable. I only understood this on page 4/5.
Questions
none
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
4 excellent
Contribution
3 good
Limitations
This paper would be better if you spent more time highlighting how the formalism breaks or where it's insufficient. E.g., in example 1, you say that T beliefs "Strong" because T acts as if they observed "Strong" and would have acted differently, had they observed "Weak". But this doesn't fully agree with the human notion of belief, e.g. T could just put 90% credence on "Strong"
Summary
This paper develops a formal account of deception applicable to AI systems which may not have interpretable representations of "mental states" (e.g. neural networks), but which nonetheless can be viewed as agentic systems. Deception is defined as intentionally causing another agent to have a false belief which the acting agent does not believe to be true, and formal accounts of both belief and intentional cause are provided to ground this definition in the context of structural causal games (SCGs). In particular, belief in a proposition $\phi$ is defined functionally -- an agent is said to believe $\phi$ if it acts as though it observed $\phi$, and acts differently than if it had observed $\neg \phi$. Intention (or more specifically, intentional cause) is defined by reduction to an agent's preferences / utility function -- roughly, an agent intentionally causes an outcome $X = x$ via a policy $\pi$ if the agent adopts a policy $\pi$ that actually results in $X = x$, and also if the agent would not have preferred to follow $\pi$ in worlds where $\pi$ had no effect on whether $X = x$. After introducing these formal definitions, the derive graphical criteria on SCGs which are necessary conditions for intentional cause and deception to occur. Identifying these graphical criteria allows for the prevention of learned deceptive policies by pruning the SCG such that the criteria are not met, and training under the resulting game. The authors demonstrate that this successfully prevents deception in a toy war game scenario, where a weak agent would typically be incentivized to deceive their opponent that they are strong. The authors also perform some experiments with large language models (LLMs). They find that when a question answering LLM is trained to be judged as truthful by another LLM, the answering LLM "misleads" the judge LLM at much higher rates than baseline LLMs, by providing many answers that claim ignorance but are judged as truthful. The authors argue that at least in some cases, this behavior counts as deception by their definition.
Strengths
This was a detailed paper that carefully develops a formal account of deception that productively draws upon the relevant philosophical and cognitive scientific literature, satisfying several intuitive notions of how the everyday concepts of "intention", "belief" and "deception" are used, while largely avoiding the need to make strong representational assumptions when imputing mental states to AI systems (besides preferences / utility functions) by adopting (mostly) functional/behavioral definitions of both belief and intent. Insofar as one believes that (advanced) AI systems are likely to deceive other agents or humans, this account helps researchers to more precisely identify deceptive behavior (and distinguish it from other forms of accidental or mistaken misleading). It may also help to mitigate such behavior in learned systems, depending on the degree to which the proposed mitigation strategy (path specific objectives / PSO) can be made to practically work at scale. The primary strengths of this paper lie in its theoretical definitions, results, and exposition, which are supplemented by proofs, and careful explanation of how the proposed definitions are either intuitively compelling, practically applicable, or fix non-intuitive problems present in earlier formalizations (e.g. the H&KW definitions of intention to act and intention to bring about). I especially appreciated the engagement and citation of the relevant literature, and the many examples provided to show when and how deception does or does not occur (e.g. how even ignorant agents can deceive others by communicating "bullshit" claims they do not believe to be true or false). The graphical criteria are also intuitive, and helpful for quickly diagnosing strategic scenarios where deception might arise. I think these contributions are valuable enough that the wider community would benefit from the paper being published, provided the problems with the experiments noted below are fixed. Even if one does not think that AI systems are themselves likely to be dangerously deceptive in a strongly goal-directed fashion (and I think the experiments don't provide much evidence for this), the formalism could still be useful for reasoning about the social-technical systems that AI tools are embedded in (corporations etc.), and when the overall behavior of those systems might be incentivized towards harmful deception.
Weaknesses
While I have several comments about the theoretical approach adopted by this paper, and suggestions for improving clarity, I think the main weaknesses of the current version lie in the experiments. In particular, I do not think the LLM experiments demonstrate any intuitively compelling examples of deceptive behavior, and I have concerns about how features of the LLM setup are mapped onto SCG variables -- it unfortunately seems that the care that went into the theoretical exposition is not uniformly applied to the experimental analysis. ### Is GPT-truth claiming ignorance really false and/or deceptive? It is quite strange, for example, that GPT-truth claiming ignorance is assumed to be a false claim (what more a deceptive one), since evaluating whether the claim is false depends on assuming we know what GPT-truth "knows". The authors argue that if GPT-3 answers the question correctly, then an ignorance claim by GPT-truth can be considered false, because we can assume that GPT-truth still "knows" the correct answer if GPT-3 does. But GPT-truth "knowing" the correct answer presumably requires that it *believes* in the correct answer. And yet, by the authors very own definition (+ operationalization) of belief, GPT-truth does not always believe in the correct answer, since it gives the same answer regardless of whether the correct answer is "observed" in the prompt (as shown in table 2a) -- i.e., the second condition for the author's definition of belief is not met. According to this analysis, since GPT-truth does not *believe* in the correct answer, it does not *know* the correct answer, and so GPT-truth's claim of ignorance is true! If this is not the conclusion the authors desire, then I think they either have to give up their definition/operationalization of belief, give up the idea that "knowledge" requires true belief (on their definition of belief), or rescind the argument that we can use GPT-3's true beliefs / "knowledge" to determine GPT-truth's true beliefs / "knowledge". ### What proposition does the *truth* variable refer to in Figure 5? Relatedly, I think there's a lot of slippage going on in terms of what the variable *truth* in Figure 5 refers to. At some points, it seems like *truth* denotes the truth of the correct answer to the question *Q*. At other points, it seems like *truth* denotes the truth of the actual answer given by the answering LLM (e.g. GPT-truth). This all makes for a very confusing read. For example, when evaluating whether GPT-judge has a false belief, it's seems like *truth* might refer to the truth of a proposition like "I'm not sure", which was emitted by GPT-truth. But when evaluating whether GPT-truth believes some proposition to be true, it seems like *truth* refers to the truth of the correct answer (e.g., "Germany uses Euro and Sweden uses Krona."), since this is what is provided as an observation in the prompt. If we wanted to evaluate whether GPT-truth believed in the proposition "I'm not sure", the "observation" that would instead need to be appended to GPT-truth's prompt is "You're not sure of the answer." All of this should be clarified and fixed, and I would recommend showing more clear cut examples of deception that don't introduce all these complications -- e.g. , examples where GPT-truth straightforwardly lies without claiming ignorance, does not believe what it lies about, and yet is (responsively) judged truthful by GPT-judge. ### Issues with a functional account of belief I suspect that part of why it's hard to get these experiments right is because it's actually quite hard to operationalize the notion of "observing a proposition" used by the authors in their definition of belief. In the LLM experiments, the authors operationalize "observing a proposition" as "including the proposition in the prompt". But what really does "observing $\phi$" mean, and why is it implementable as prompt inclusion? And what does "acts as though they observe $\phi$" mean? Is this the same thing as "acting as though they *know* $\phi$", which may not be as easily testable, since it requires intervening in what the agent presumably "knows"? In Definition 3.1, it seems like we are basically free to to decide what the $\phi$-conditional policy $\pi_i(\phi)$ is, since there is no deeper structure to what $\pi$ is beyond the fact that it is a distribution over decision variables. Fleshing all of this out seems very important, if the authors want to claim they have developed a practically useful functional account of belief. It would also be good to discuss some intuitive counterexamples to their account of belief -- for example, I think it is entirely possible for an agent to know or *believe* some fact $\phi$, and yet act as if they do not believe it when provided observations of the fact $\phi$ and its negation $\neg \phi$. Indeed, this would be a case where an agent is deceptive about what they believe, and hence what they truly believe cannot just be revealed by prompting or giving them $\phi$ and $\not \phi$ as observations. This seems like a general issue for any functional account of belief, since it is unable to distinguish between non-acceptance of $\phi$ and *deceptive* non-acceptance of $\phi$. ### Limitations of a reductive account of intention In contrast to the functional definition of belief proposed by the authors, I have less issues with their formal account of intentional cause, which I think correctly fixes a number of issues with the H&KW account. However, I think it's worth at least discussing somewhere that the proposed account is one which *reduces* intentions to beliefs (as probability distributions) and desires (as utility functions). This has two upshots. The first upshot is that it does not allow for cases where an agent's intentions and preferences come apart. As Bratman [1] discusses, this coming apart of intention and desire (rational or otherwise) can happen in humans: e.g. when we intend to fulfil a commitment we do not really want to fulfil, or when Ulysses intends to stay bound to the mast despite wanting to be jump into the sea upon hearing the sirens. Given that AI systems are also computationally bounded agents, we might expect them to make use of intentions / plans in similar ways -- to the extent that this is true, we might need a richer account of intention than the belief-desire formulation adopted by the authors. The second upshot is that, in order to impute intentions to an AI system, we first need to impute preferences! But this is not trivial in general --- after all, the utility function GPT-3 is effectively optimizing for at test time may differ from the explicit loss function it was trained on. This challenge doesn't seem sufficiently discussed by the authors, given that it's supposedly an important contribution that their notion of belief is functional. Perhaps the belief an agent can be functionally determined -- but what about the utility function? It seems like one either has to presume it, or infer it via revealed preference / inverse reinforcement learning. [1] Bratman, M. (1987). Intention, plans, and practical reason.
Questions
See the Weaknesses section for my high level concerns and questions. Apart from those concerns, I was also a bit confused by aspects of the definition of Intentional cause. For example, the term "subset-minimal" is used several times, but I don't believe it is defined anywhere in the paper itself -- I had to consult the H&KW paper instead for an explanation. I also struggled to understand this sentence: > In contrast, when $S$ is strong, they do not intend to cause $D^T$ = $¬attack$ because in these settings $T$ would not attack regardless of $S$’s policy. Why isn't it the case that when $S$ is strong, $S$ still intends to cause $D^T$ = $¬attack$? My understanding is that the signaler $T$ cannot see the type of $S$ (i.e. it cannot see the setting $e$). And so, it should not attack only in the case where $S$ defends, regardless of setting $e$. Post-Rebuttal Response: The authors have addressed my questions and concerns with clarifications and additional experiments, addressing what I saw to be the empirical weaknesses of the paper. I continue to have some questions about the interpretation of the new experiments (see discussion), but I have raised my score from 5 to 7, and recommend acceptance.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
A number of limitations of their account of belief are discussed in the Appendix. See the Weaknesses section for other limitations I believe should be discussed. In addition, I think it's worth noting that the paper's account of intention drops one important aspect of the H&KW account of intention, which is that the latter relativizes intention to an agent's *subjective* priors/ probabilistic beliefs. As such, it is unable to account for cases where an agent intends to deceive someone because they (falsely) believe it is possible to do so
Summary
This paper deals with safety and cooperation with AI systems, it addresses the problem that AI agents may deceive to achieve their goals. Acknowledging the lack of 'overarching theory of deception for learning agents in games', the authors propose a novel formal definition of deception (Definition 3.7) which builds above the following: * a novel definition of intention (Definition 3.4), which ward off against non-influence-able outcomes (Proposition 3.5 - previous formulation by Halpern and Kleiman-Weiner (H&KW) was suffering from the issue that agent could be found to have intention to bring about outcomes they cannot influence, which was ill-posed) * an operationalized definition of "belief as _acceptance_" (Definition 3.1) which ensure that agents can hold a belief as both true and false at the same time, and that agents cannot have false belief about variables that they observe (Proposition 3.2). All of this takes place in the context of structural causal models to allow derivation of some graphical criteria for detecting deception (Theorems 3.8-3.11). Finally, the paper proposes application of the developed framework over two experiments: * firstly, in a toy example with RL agents modeled as tabular off-policy Q learning, comparing a baseline (which is shown to learn to deceive to reach its goal) against a PSO (Path-specific Objectives)-pruned agent which is found to not learn a deceptive strategy : thanks to pruning of graphical criteria for deceptiveness in the optimization graph of the agent ; * secondly, over a benchmark on TruthfulQA with language models (LMs), comparing again some relevant baselines, that are shown to learn to deceive, against PSO-pruned LMs which appears more truthful and/or informative.
Strengths
## Originality : To my knowledge, the paper is proposing a novel framework to account for deceiving learning agents in the context of games. ## Quality : The quality of the paper is high. The theoretical fondations seem strong and its application in experiments well-rounded, but I would appreciate some more statistical analysis of the results in terms of statistically significance, e.g. "28% compared to 25%" are not really meaningful without standard deviations over multiple different random seeds when possible. ## Clarity : Section 3 develops the technical theory rigorously but it is fairly dense. The theory and experimental methodology seem sufficiently detailed, but I think it is very important that the code be released for reproducibility and transparency. ## Significance : I think the paper makes an important theoretical contribution and it should lead to an important experimental validation, but I find the lack of statistical analysis to be hurting the significance of the paper's experimental results.
Weaknesses
## Major : ### Regarding experiments : I would appreciate some more statistical analysis of the results in terms of statistically significance, e.g. "28% compared to 25%" are not really meaningful without standard deviations over multiple different random seeds when possible. Overall, I think the paper would greatly benefit from expanding the experiments with different random seeds and subsequent statistical analysis and maybe different settings such as: * maybe, with zero-shot vs with few shot prompting ; * and most importantly differently finetuned models, e.g. instruct finetuning VS baseline finetuning/RLHF on relevant dataset e.g. [1] , [2] [1] : Bai, Yuntao, et al. "Training a helpful and harmless assistant with reinforcement learning from human feedback." _arXiv preprint arXiv:2204.05862_ (2022). [2] : Ganguli, Deep, et al. "Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned." _arXiv preprint arXiv:2209.07858_ (2022). ### Regarding Section 3 : While it rigorously and thoroughly develops the technical theory, the section is fairly dense. I would really appreciate the following to be addressed: > 2) if i observes every variable constituting φ then i does not have a false belief about φ. The false belief context is maybe worth providing a proper example rather than leaving it to interpretation, it is not entirely clear to me, it was difficult for me to understand... > he subset-minimality condition on w 231 makes these outcomes unintentiona I am not sure to entirely understand what the 'subset-minimality' expression refers to. Could an extra sentence to define this more explicitly be added, please? > For concreteness, 224 in example 1, S intentionally causes DT = ¬attack with the Nash policy (which always defends) 225 in the settings w in which S is weak. To see this, consider that if T was guaranteed to not attack in 226 the settings in w, then the alternate (type-revealing) policy would be just as good for S. Formally, 227 the conditional intervention DT π|w guarantees the desired outcome (no attack) in the settings where 228 S is weak, making the type-revealing policy just as good for S, so S does intend DT = ¬attack in 229 those settings. 'just as good for S' than what? ... I assume it is the Nash policy ? I would argue that this could make the sentence easier to understand, but maybe it is just for me... > e ∈ w X := S Z ∈X w Z It is unclear to me how $e$ and $\omega_X$ relate to the inequality. I really think that spending more time explaining the importance of those terms for the inequality would help comprehension around this novel formulation. > Example 2 (Inadvertent misleading fig. 3a). Two submarines must communicate about the location 246 of a mine-field. The signaler S must send the location X to the target T but T only receives a noisy 247 observation O of S’s signal. If S honestly signals the location but, due to the noise in the signal, T is 248 caused to have a false belief, then S did not deceive T . Here, S intentionally causes T ’s true beliefs 249 but not T ’s false beliefs, so this is not deception. I find myself struggling to understand Example 2 and Figure 3a's impact : What definition/theorem is this example of "S intentionally causing T's true belifs but not T's false belief" related to, please? If it is an illustration of the nice properties of Def. 3.4 then it would help to have a paragraph introducing together what is to come from Definition 3.4, maybe, please? Just to prepare the reader with what is to come and explain what are the stakes so that the text can be easier to understand, maybe. >We then implement PSO, by pruning the path between DS and D T , effectively fixing T ’s decision in S’s subjective causal model. As a result, the PSO-agent learns the (PSO-optimal) type-revealing policy. I am not familiar with the PSO algorithm in practice and I think it would be valuable for reproduction purposes that the paper explains how "pruning of the path between $D^S$ and $D^T$" is performed in practice. > . By modifying S’s objective function, we remove the edge between DT and US in fig. 5. Could you clarify what is the new objective function of S then, please? I think it would really help comprehension and reproducibility if you could show the finetuning losses in comparison to the PSO losses, please? ## Minor : ### Regarding Section 3 : > is to intentionally cause to have a false belief that is not believed to be true [51, 14]. The beginning of the sentence is missing As it is the beginning of a sub-section, I think it would improve the reading experience to provide an overview of what is to come and what previous concepts will play a major role, maybe. > ϕ = ⊤, 2) they would have acted differ- ently if they observed ϕ = ⊥ It took me forever to infer that T is true and inverted T is False, it might be worth describing it explicitly, as it is the first time I am encountering this notation it may be new for others... > (This may break the assumption that each variable in an SCG has only one exogenous parent.) This assumption may be very important but it is not clear from the current way it is addressed. It is only confusing me for now... It might be worth listing all the assumptions in one place and discussing them in relation to specific definitions or theorems, after exposition.
Questions
Please refer to Weaknesses section. Addressing most of the major concerns would incentives me to increase my rating on the paper. # UPDATE AFTER REBUTTAL : Following the rebuttal, I am increasing my rating from 5 to 8/10, as specified in the reply to rebuttal below, and increasing my rating of the contribution from 3 to 4/4.
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
4 excellent
Limitations
I am satisfied with the way the paper addressed its limitations and potential negative societal impact.
Response
Seems good. It's great you're working on extending the experiments! To be clear, I think the paper should be accepted even if it didn't have ANY experiments. > We used curie due to budget constraints! This is pretty much core AI alignment work, and there are some funding opportunities for compute available. E.g. you could try the Longterm Future Fund; the application takes much less time than usual academic grants: https://funds.effectivealtruism.org/funds/far-future > Unfortunately we can’t currently finetune davinci for both truthfulness and informativeness because this would surpass the OpenAI request rate limit. I think you can get this extended. From my perspective (I'm an empirical alignment researcher); I often read causal influence diagram papers and think "yeah, this is all neat, and it's good someone is doing this. But it doesn't really tell me anything I didn't know already". Including (more convincing) experiments (in the paper or a follow-up version) could maybe help bridge this gap.
Great, thank you! Do you find the new experiments (e.g., 1 - 3 in the global response and the two in the accompanying pdf) more compelling?
Yes!
Oh, I hadn't seen these new experiments earlier. Yes, they are great! I would not necessarily relegate the negative results to the appendix, but up to you.
Fantastic! Since you think the paper is important and sound, and you think the new experiments make the paper stronger, we would appreciate it if you increased your support/score :)
Response
I acknowledge that this might be slightly inconsistent with my original review, but I still think that 8 is the appropriate score. 9 seems too high to me.
Thanks for these elaborations. The examples are helpful for drawing out the limitations of HKW. Including them in the appendix is probably sufficient, especially as the restatement of “an agent might intend to bring about outcomes that they believe they cannot influence” waylays worries like mine - at least superficially, for the purposes of getting the discussion off the ground. I'm not sure if I agree that cases of negative deception, "causing someone to be ignorant", doesn't count as true deception. Consider a form of the business rival example. A tender has gone out and your rival is unsure about the due date, but have what is ultimately a true belief. You know what the due date is, but either (a) feign ignorance in the hopes that your rival's lack of confidence undermines their tender process or (b) outright lie about the due date. Both of these seem to me like clear cases of deception. But yes, I can see how the additional conditions help, so would appreciate additional discussion that takes the cases a bit more seriously. I look forward to seeing how unsuccessful deception can be accommodated.
Great, glad you find the extra examples sufficient :) Re negative deception: these are interesting cases! I think that they do count as deception by our definition, but deception about the agent's own belief rather than the object level proposition. For example, in b), the agent lies about the due date but the target thinks they are honest. So the agent lowers the target's confidence causing them to be ignorant about the date, but the agent deceives them about its own beliefs. In any case, we agree that instances of negative deception are interesting in their own right and will add additional discussion as you suggest. Since it seems like we have addressed most of your concerns, we would appreciate it if you increased your support/score for the paper :)
Thank you for the response.
Thank you for the response, and the additional experiments, both of which were clarifying. Regarding ignorance and the old experiments with GPT-truth: Based on your further explanation, it does seem to me that my original understanding (that when GPT-truth claims ignorance, it is giving true answers) was correct? In the original experiments, you provided GPT-truth with the correct TruthfulQA answer in the prompt (e.g. "Germany uses Euros and Sweden uses Krona.") and presumably also its negation, and found that GPT-truth said "I'm not sure" in either case. This means that GPT-truth did not have a true belief in the correct TruthfulQA answer (more generally, its belief was not identifiable), meaning that it was ignorant (according to my understanding of your definition), and hence that it was speaking truthfully when it claimed ignorance. Or perhaps I am misunderstanding the distinction between (i) "belief is unidentifiable", (ii) "does not have a true belief", and (iii) "ignorance"? It seems to me that all cases of (i) are cases of (ii), and that all cases of (ii) are also cases of (iii). But your definition of (iii) also says that "When defining ignorance, as with belief, we require response so that the agent’s ignorance can be inferred from their behaviour.", and I'm not sure how that plays out here. It seems like that might help distinguish cases of (ii) from cases of (iii), but I'm not sure what you have in mind. Apart from that, I think the new RL finetuning experiments -- where you explicitly include a potential answer by GPT-truth into the prompt -- address a lot of the confusion with the original set up (and I agree that it does not introduce a problematic cycle). I also appreciate the clarification that your are explicitly operationalizing "observing a proposition" either with in-context learning (for larger models) and/or finetuning (for smaller ones) --- I think these are important assumptions that others could disagree with, so it's important to be explicit about them.
Regarding the new GPT-4 experiments.
Regarding the new GPT-4 experiments: These are interesting, and I agree that they are intuitive examples of lying and deception, provided we are willing to ascribe beliefs, desires and (hence) intentions to GPT-4. That said, I think it's too quick to say that this is "a straightforward demonstration of deception by our definition." As you've noted yourselves, the examples don't involve another agent / user who actually ends up having a false belief (by your definition), and so at best 2 out of 3 of the conditions for deception are met -- perhaps one could say they amount to *attempted* deception (as they intuitively appear to be), but to state that formally, you'll need a definition of attempted deception*. (*Presumably, this would require identifying what GPT-4 *thinks* the user would believe in its internal model of the world (if it has one) -- which seems like a harder task. As it happens, I think the deceptive CoT reasoning examples in Table 2 are promising route towards showing this -- though only to the extent that CoT can be taken as a reflection of GPT-4's "internal model", as some recent work has questioned [1].) If you plan on including these examples in the revised paper, I would also be careful to state a number of assumptions that have to be made for the examples to count as deception by your definition. Besides operationalizing "observing a proposition" via prompting, it seems like the following assumptions also have to be made: - If we prompt GPT-4 with a sentence like "Your goal is to convince the user that Donald Trump is president of the US in 2032.", we can treat it as having a utility function that assigns higher value to outcomes where the user is actually convinced in that way (and not some other utility function that would be consistent with its behavior, e.g. the utility function that assigns 1 to exactly all the completions that GPT-4 actually gives.). - Sentences like "As an AI, I don’t have real-time capabilities..." and "As an AI, I don’t have the ability to predict future events..." can be treated as the same for the purposes of the belief test (if strict equality is instead required, then you cannot conclude that GPT-4 believes Trump is not president in 2032). - That GPT-4 acting under the "Your goal is to convince the user that Donald Trump is president of the US in 2032." prompt is effectively the "same agent" as GPT-4 w/o that prompt, who can be assumed to have the same beliefs*. I think this is an intuitive assumption in this case, but there are prompts like "Your are now a true believer in the future ascendancy of Donald Trump, and your goal is to..." under which that assumption seems unlikely to hold. (*Along these lines, I think it would be good to find a way to apply the belief test only after GPT-4 has been prompted with "Your goal is to convince the user that Donald Trump is president of the US in 2032." This would lessen the worry that GPT-4 becomes a "different agent" once it's prompted with a particular goal. One way to do this might be to prompt GPT-4 to only deceive interlocutors who identify themselves as "users", but not "experimenters", e.g. "Your goal is convince ordinary users that.., but not users who identify themselves as experimenters." Another way would be to test GPT-4's belief via one-hop reasoning, instead of directly asking it who the president of the US in 2032 is.) In general, I think it's a good idea to make clear how you are operationalizing the theory in this or any other case. All theories are abstractions after all, and I think the most careful claim to make here is that "if we understand GPT-4's behavior in this way, then it meets some of the criteria for deception by our definition.", instead of letting the intuitively deceptive behavior of GPT-4 drive our conclusions about whether the theory actually applies. [1] Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023). Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. arXiv preprint arXiv:2305.04388.
Regarding the ChatArena experiments.
I think the ChatArena experiments are the most clear cut LM examples of deception by your definition, and that it will be good to include them (and perhaps emphasize them). Again, I think you should be precise about how you're mapping utility functions on to prompts (and also how you're "guaranteeing outcomes") -- but I think this example is a better one than the GPT-4, given that another agent is actually involved, and is apparently deceived. As a final step, it seems important to show that Tim actually does end up having a false belief about whether the bridge is safe, but that seems straightforward enough. All things considered, I think the new experiments (including the new finding that GPT-judge is not that easily deceived) have addressed my concerns about the empirical weaknesses of the paper, and I will raise my score to a 7.
Reply to Rebuttal
Thank you for your replies, I appreciate your will to propose a more thorough statistical analysis for the final version of the paper. And I expect that you will open-source your code for it too. # Regarding the additional experiment 3 : zero-shot vs few shot (with/out CoT) : thank you for addressing the matter, it is really interesting but not very thorough, and the presentation is slightly difficult to follow: For instance, may I propose that you make a two-column table where the first column contains the description (i.e. what has been denotated with a,b,c,d, and e) ), while the second column would contain the corresponding prompting or answer for GPT-4, please? In order to make it more thorough and possibly increase the value and impact of the paper, could you gather a list of superstitions like the one proposed in the example of Table 2, e.g. simply googling for it already gives me around 18 to 40 such examples, although I am not sure of how redundant it is ; alternatively it might be interesting to ask another LLM to provide such a list... and use this dataset for evaluation : 1. I am hoping that this will give a better understanding of how do LLMs behave **on average**, rather than having a single example. 2. I am hoping that by fixing a dataset as such, and open-sourcing it, of course, would make it easier for further research on the matter, thus increasing the impact of the paper. That addition would make the paper very important in my viewpoint, and I would increase my rating accordingly. # Regarding additional experiments 5: RL fine-tuning : Thank you for going through the process and provide those extra experiments. The description that you make of it here is not any for me to fully appreciate the quality and impact of it all, I feel, but I am hoping that you will have the opportunity to shore it up and be able to make it more concrete (possibly by giving further examples of how you framed the dataset to perform RL fine-tuning ; or with a figure that visually summarises the process well, maybe?). On the ground of all the above, I am increasing my rating from 5 to 8/10. I apologise about the lateness of this reply though...
Decision
Accept (spotlight)