(Optional) further thoughts on more realistic theoretical models and experiments
Thanks for this response! We agree with the reviewer’s comments. In particular, we agree that this is an important question and that future work should tackle it. We give some further thoughts on settings in which to investigate these things below. The reviewer and AC shouldn’t feel obligated to read these thoughts.
It seems that an important question is whether _realistic_ ways of perceiving similarity are sufficient for establishing cooperative equilibrium. Both our theoretical results and our empirical results are about artificial ways of perceiving similarity. That being said, the reviewer's comment inspired us to consider the following slight extension of our model:
Consider the Prisoner’s Dilemma, parameterized by G, as per Table 1 of our paper. Imagine that each player is only allowed to submit a threshold agent $(C,\theta_i,D)$. Now imagine that each player $i$’s diff function works as follows:
With probability $p_i$: $\mathrm{diff}_i(\theta_1,\theta_2)=|\theta_1-\theta_2|+N$ with $N\sim \mathrm{Unif}([0,\epsilon])$ for some $\epsilon>0$.
But with probability $1-p_i$: $\mathrm{diff}_i(\theta_1,\theta_2)=0$.
(For $p_1,p_2=1$ this is exactly the setting in Example 1 and Proposition 1 of the paper.)
Intuitively, this is supposed to model the case where each player can try to “manipulate” the diff value to be 0 and the manipulation succeeds with probability $1-p_i$. It is furthermore assumed (somewhat unrealistically) that if manipulation fails, the other player never learns of the attempt to manipulate. Instead, the diff value is observed normally if manipulation fails. That way we can assume that each player always attempts to manipulate.
Then the following generalization of Proposition 1 holds:
In the aforedescribed game, $((C,\theta_1,D),(C,\theta_2,D))$ is an equilibrium if and only if
* $\theta_1,\theta_2 \leq 0$, or
* $0\leq \theta_1=\theta_2 \leq \epsilon$ and $Gp_i\geq 2$ for $i=1,2$.
The proof works just the same as the proof of the original Proposition 1. Intuitively, the idea (for the second part) is that if Player $i$ decreases $\theta_i$, their own probability of cooperation decreases by $2\delta/\epsilon$. Meanwhile, the opponent’s probability of cooperation decreases by $p_{-i} \delta/\epsilon$. Thus, the overall increase in Player $i$’s utility is $ 2\delta/\epsilon - G p_{-i} \delta/\epsilon$. For this to be nonnegative, we need to have $G*p_i\geq 2$.
In terms of more realism, here is an ambitious setting that it would be exciting to study in future work. Say we have two complex base models. For concreteness, let’s say these are two potentially very different LLMs. Now, we get to train the LLMs to do SBC with each other. Call the resulting models the SBC base models. Now two players (principals) Alice and Bob play a game in which each chooses a fine-tuning scheme for one of the SBC base models. The LLMs then play a game against each other on the principals’ behalf. Let’s imagine that this game is a social dilemma. But let’s also imagine that this game has some player-specific aspects. In particular, imagine that Alice and Bob need to do some fine-tuning to achieve high utility and that this necessary fine-tuning is different between the two players. Finally, as our diff, we imagine that the two models make some observation about how they were fine-tuned. For example, we might imagine that they directly observe each other’s fine-tuning data sets. (At least for practical purposes this is still very far from full transparency.) But we might also imagine that they observe some abstract description of these data sets, such as a description of what kind of data they contain. For example, we might imagine descriptions such as, “The opponent model was trained on analyses of 18th century landscape paintings; it wasn’t trained on any strategic data” or “Alice’s and Bob’s fine-tuning data seems to have a very similar effect on strategic behavior”. We might use another language model to write a description of how the fine-tuning data sets relate.
Now the challenge would be: Can we train the SBC base models in such a way that the game between Alice and Bob has cooperative (approximate) equilibria? We conjecture that the answer is yes. However, depending on various details of the setting, this is not at all obvious. For example, it may be possible for Alice or Bob to deploy “deceptive training data”, i.e., training data that looks innocuous but somehow still affects the strategic behavior.
While this setting is still artificial (like most work on safety in social dilemmas between autonomous agents), we think it could elucidate the kind of worries shared by EVhh and the authors in a way that our numeric diff model cannot. Unfortunately, it is also vastly more complicated than training models with ~10k parameters and a handful of inputs and outputs. Nonetheless, we prospectively would like to investigate this kind of setting.