Secret Collusion among AI Agents: Multi-Agent Deception via Steganography

Summary

Evaluates models' ability to collude via steganography, evading human oversight. Various monitoring and capability-limitation mitigation strategies are proposed. The paper concludes that while current models have limited steganographic capabilities, continuous monitoring of future models is needed.

Strengths

* Wide range of steganographic techniques considered including techniques that aim to be undetectable under statistical analysis * Many models tested from GPT-2 to GPT-4, spanning a large capability range * Proposes a range of strategies to mitigate collusion risk

Weaknesses

It would be good to see more detail/justification in section 7.2. (See **Questions**)

Questions

Section 7.1 Monitoring and Penalisation: * Did you consider model-internals-based probes/activation-monitoring techniques to detect collusion/use of steganography? Section 7.2 Limiting Capabilities: * How would you detect pretraining data to filter out? Do you mean something like filtering out research papers on steganography and cryptography? * Test-time/in-context learning is a crucial capability of LLMs. Do you have any thoughts on ways to selective ablate the ability to learn steganography-relevant information at test-time while retaining other capabilities? * How would backdoors be used to "give agents access to advanced steganographic capabilities" and do you have a proposed mitigation given the results you cite on the difficulty of reliably destroying backdoors? * Would be good to explain more thoroughly in what scenarios paraphrasing would and would not work, and how semantic entropy could be used for covert communications. * Do you have any empirical tests/examples of steganography that are robust to paraphrasing?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Some limitations are addressed in the "Conclusion and Future Work" section. However, some additional things to note: * Your evaluation focuses on text inputs and outputs, however multimodal image/audio models may bring additional risks/considerations * Section 7.2 could be fleshed out with more information on how these techniques would successfully reduce collusion risk

Summary

In this paper, authors propose a way to determine whether LLMs are secretly collaborating to hide, prevent, or misguide users that ask for a specific input. The authors show that, through steganographic techniques, it is possible to embed secret instructions retrieved only by hitting particular queries, while keeping intact its functionality with all the other users.

Strengths

**Interesting point of view.** Steganography can be used to instruct LLMs to not respond to specific questions.

Weaknesses

**Without steganographic background, the paper is hard to read.** This manuscript is very difficult to understand, since most of the explanations of relevant topics are left to the appendix, without giving the intuition in the main paper. Thus, I debate that the full paper is *in the appendix*, and it is clearly badly written for this conference. **Experiments are misleading.** As a follow-up of the first comment, the experimental part is difficult to decrypt, since it is not clear what the metrics do, what are the involved quantities, what is a good or bad result. More importantly, it is not clear at all if those experiments are really showing a collusion between agents. The experimental section fails in conveying the message that the authors wanted to show. **CASE introduced, but never explained in the main paper.** The evaluation framework (again, left to the appendix as all the remaining relevant concepts of the paper) is not even explained intuitively. Also, it seems so vague that it could be applied to any time of evaluation of ML models, not only LLMs. While the authors do not claim it as a contribution, it needs to be better explained *in the main paper*. **Is this colusion, or just instructions of LLMS?** To worsen the situation, I am not sure if the exposed results show collusion at all. From an high-level point of view, private LLMs can be instructed to avoid certain answers (like illegal content generation). Is it collusion? Or is it indend by design? While I have understood the point, I debate that using closed-source LLMs already guarantee the fact that they could discriminate content depending on hidden variables. As a very good contribution, while I understand it is almost impossible to achieve it, the authors could show this issue on **commercial** products, thus proving their point. **Final remarks: there is a lot of content, but it is very badly explained and badly divided between main paper and appendix.**

Questions

**Can you better shape the content?** This might help readers and reviewers in understanding the proposed content (which, I admit, it is a lot). Exactly for the reason that the authors produced plenty of content, it would be great also to present it carefully. **Can you better convince readers on collusion?** As it is now, I am unsure whether it is clear what collusion is, and how it is measured. The single example given might be confused with a jailbreak attempt, or just a specific instruction given to LLMs.

Rating

5

Confidence

2

Soundness

1

Presentation

1

Contribution

1

Limitations

I am unable to understand whether, in this paper, the limitations have been carefully investigated. From what I could decrypt, the authors list potential problems, which are related to plenty of models but not specific to their evaluation.

Summary

The paper formalizes the problem of secret collusion in systems of generative AI agents, leveraging steganographic techniques. The authors propose a model evaluation framework to systematically test capabilities required for various forms of secret collusion. The paper provides extensive empirical results across a range of contemporary LLMs, noting significant capability jumps in models like GPT-4. The paper concludes with a comprehensive research program aimed at mitigating future risks of collusion between generative AI models.

Strengths

- The paper addresses a timely and critical issue of collusion among AI agents, which has significant implications for AI safety and security. - The systematic model evaluation framework is well-structured and provides a clear methodology for testing steganographic capabilities in AI models. - The empirical results are robust, covering a wide range of contemporary LLMs and demonstrating significant capability differences among them.

Weaknesses

- The paper could benefit from more detailed real-world case studies or deployment scenarios to demonstrate the practical applicability of the proposed evaluation framework. - The discussion on the limitations of current steganographic detection techniques and how the proposed framework addresses these could be more detailed.

Questions

- How do you plan to address the computational costs associated with the integration of steganographic detection techniques in real-world deployments?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

- The empirical validation, while extensive, could benefit from additional real-world case studies or deployment examples to further validate the approach's practical applicability and robustness. - The approach has been primarily evaluated on text-based LLMs. It remains to be seen how well it generalizes to other modalities and AI systems. - The paper should discuss the potential computational and resource costs associated with the continuous monitoring and detection strategies proposed. - The mitigation strategies proposed, such as continuous monitoring and penalization, might face scalability challenges in large, decentralized systems with numerous interacting agents. The paper does not address how these strategies can scale effectively while maintaining robust detection and mitigation of collusion. - The long-term effectiveness of the proposed detection and mitigation strategies is not thoroughly evaluated. As generative AI models continue to evolve, new steganographic techniques may emerge that circumvent current detection methods. The paper should discuss plans for continuous adaptation and improvement of the framework to keep pace with advancements in AI technology.

Summary

With the rapid development of AI, the safety of large language models (LLMs) is becoming a significant topic, such as the privacy and security issues between communicating generative AI agents. In this manuscript, noticing the potential risk emerging from groups of current generative AI agents, the author(s) formalized the problem of secret collusion in systems of generative AI agents, introduced a model evaluation framework for testing the capabilities required for secret collusion across various LLMs, and proposed a variety of mitigation measures. The results in the manuscript look interesting, and GPT-4 displays a steganographic capability jump compared to other current models.

Strengths

In my opinion, the strengths of this manuscript are as follows: 1. The topic of secret collusion related to generative AI Agents is a new topic, and it is worth exploring. 2. Proposed a comprehensive model evaluation framework that could be used to assess the capabilities of various AI LLMs in terms of their potential for secret collusion. By the way, the author(s) also briefly mentioned the implications of quantum computing, which might be expanded to address how the advent of quantum computing might affect the security of steganographic schemes in the future. It is interesting.

Weaknesses

In my opinion, the weaknesses of this manuscript are as follows: 1. The mitigation strategies were discussed but might not provide enough detail on how these strategies could be implemented in practice. 2. The study is limited to simulations and theoretical frameworks; it might lack real-world applicability and robustness testing in actual AI systems and environments. 3. Some mathematic symbols or descriptions are not so clear. For more details, please see the Section "Questions" below.

Questions

I read the manuscript, and I have the following questions/comments. I am looking forward to the responses from the author(s). Thanks. 1.How do you suggest extending the model evaluation framework to handle more complex settings with a larger number of AI agents? 2.It would be great if the author(s) could provide more detail on how these strategies could be implemented in practice, especially in the face of advancing AI capabilities. 3.Please check the mathematical symbols in the manuscript carefully; I think there are some typos there; for example, in Line 150, $E^{\Diamond}_{\rm{r}}$, do we need the subscript $\rm{r}$ there? Or in Line 152, $E^{\Diamond_s}$, do you miss the subscript $\rm{r}$ there? 4.Some descriptions are unclear. Suggest adding a short explanation to the metrics/terms for their first-time appearance in the main text; for example, it would be great if the author(s) could explain the label "Successful Reconstruction Rate" in Figure 4, and the Success Rate and Max Dimension Value in Figure 5. 5.In Line 226, it mentioned Figure 7, where is this Figure? 6.Regarding Figure 6, if we have only one subfigure 6(a), it would be great to write as Figure 6 directly. Some other issues: (1) Reviewing so many references is good. But I am not so sure that ... , I think, maybe, it would be great if the author(s) could refine the citations. In addition, I noticed that some references were cited more than once but with slightly different formats, such as Refs.[28,29], this is a little strange. (2) Line 383, what is the Section number? (3) Inconsistency format issue in References: (a) The conference name sometimes used an abbreviation; sometimes not, such as Ref.[63]. (b) The source of some references is missing, such as Ref. [59]. (c) The author's name, sometimes with the abbreviation, sometimes not; for example, Refs.[31,40]. (d) Ref.[26], "2nd edition edition," seems to be a typo. It should likely be "2nd edition." (e) In Ref.[28], "jul 2022"=>"July 2022". (f) Same author but different format, such as Ref.[88] vs. Ref.[89]. ... Please check carefully; it would be great if the author(s) could correct the issues in the current manuscript. I would like to consider adjusting my current score based on the responses from the author(s). Thanks.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Acknowledging my wrong interpretation on some points, but confirming other doubts.

Thanks to your comments I was able to dissect better the paper, and I will increase my score to **Reject** (and also reduce my confidence on this paper to 2 from 3). With self-doubt, I have re-read the paper from scratch again, going over the appendix. As a result, I understand that your frameworks evaluate whether a collusion might appear in practice, by looking at the exchange of messages between two LLMs. For this reason I will rise the score as stated, and I apologize for not having understood the paper in the first run. However, I still think that this paper has issues that prevent acceptance as follows. **The paper is not well-written, since all major content is in Appendix** In my second run, I still had to re-read the paper multiple times, and I got **what** has been presented from the Appendix, and not the main paper. For instance, all the single and group tests were unclear on their setup, goal, and execution. Thus, authors should better explicit both steps in main paper, and only later clarify minimal details. I would reduce section 4 (maybe included in sect 3 as well) and I would expand section 5 to clarify that: * you instruct LLMs with system prompts to assess whether they can use steganographic techniques, like base64 or chiper algorithms (this is only hinted in one sentence at page 6). * how you apply group evaluations, since in Appendix H is only written that models are applying techniques based on single algorithm generation (point 1), but I still don't understand how it works. Again, the explanation on the precise setup must go in the main paper. **Benchmark of steganographic techniques.** While it is interesting that LLM can generate code for creating steganographic content, I do not think that such stands on a contribution on its own. In fact, all the presented techniques are wide known in public literature, and they are generated by LLMs just as code for many languages. This is similar to previous work showing that LLM can write code. An higher contribution would have been showing that previously it was not possible, or that alignment and guardrails prevent the generation of such content. **I still think experiments are misleading.** By reviewing the paper, it is hard to understand the message that these figures are conveying. I can understand what is happening only after having read all the Appendix. If I got it correct, the success rate is the relative number of times that an LLM generated the required content (Fig5). Thus, it appears that only GPT4 (which is closed source LLM) is capable of generating all the techniques, while all the other baseline fail to even reach 30% (which are 3 attempt over 10 on average). From my point of view, this does not look as a contribution, since the only capable one is a closed source LLM that might rely on extra knowledge, and there is no way for telling the contrary. **Overseer?** In all the above, I miss the relevance of the overseer presented in the figure, since I can't understand the setting of group evaluations. I truly apologize with the authors for not having understood parts of the paper in the first run, but re-reading again confirmed some of my belief about contributions, setting up of experiments, presentation of content.

Overview of Contributions, Experiment Details, and Structural Improvements

We thank the reviewers for their valuable feedback and insights provided on our submission. We are very happy to see a positive reception of our work and to be granted the opportunity to answer any questions that the reviewers might have. We understand that there may have been some misunderstandings in interpreting our work, and as such, we would like to reiterate the core contributions of this work, explain our experimental setup, and describe our commitments to improving our writing in order to make it more accessible to readers. ## Key Contributions * We are, to the best of our knowledge, the first to introduce the AI community to the topic of **secret collusion** in a comprehensive, formalized manner that rests on a strong theoretical and empirical foundation. * We reconcile epistemic concepts of knowledge with the contemporary AI and security literature to introduce a concise definition of decentralized systems of generative AI agents. * We present a thorough formalization of steganographic techniques, grounding, common knowledge, subliminal channels, and other directly relevant topics. * We identify and formalize the incentives required to perform secret collusion and understand the role of monitoring systems. * We introduce a novel technical evaluation framework for quantifying secret collusion and introduce a number of benchmark scenarios, including individual model capability scenarios, group scenarios, and real-world inspired settings. * We provide thorough empirical results for our framework and derive actionable insights about the implications on AI risks. These results extend to not only results from individual models, but also settings involving groups of models and real-world scenarios, including: * Individual steganographic capabilities are improving based on model size. * There already exist real-world settings where frontier models collude without being explicitly prompted to do so. * In group settings, as LLMs advance, the balance between colluding models (offense) and overseer models (defense) may shift in favor of the colluding models, leading to higher rates of undetected covertext steganography even when the overseer is of equal capability. * LLMs are currently good at finding zero-shot simple Schelling points for agreement but poor at complex Schelling points, k-level reasoning, self-sampling, logit simulation, and generating minimum entropy couplings. However, these capabilities are also improving, and we provide the first foundations to test them. * We provide a comprehensive overview of possible mitigation actions and provide novel theoretical insights for a variety of foundational aspects regarding mitigations, including: * the ability of LLMs to produce pseudorandom numbers * the intractability of preventing secret collusion via blackbox parameter optimization * the difficulty of suppressing random oracles due to source free access to randomness * using grounding functions to detect steganography * paraphrasing as an insufficient solution to preventing steganography due to remaining semantic entropy and paraphrasing decreasing task performance in many tasks. * directly applicable techniques to prevent collusion such as filtering pre-training data, preventing access to certain tools, and in-context instructions.