Summary
This paper presents an empirical analysis of an interesting idea wherein combining frontier and weaker large language models, which are claimed to be safe individually, can be used to accomplish harmful tasks, such as generating vulnerable code, explicit images, and manipulative tweets, through either manual or automated task decomposition. The authors argue that such model combinations substantially increase attack success rates, highlighting the need for red-teaming efforts that go beyond evaluating individual models in isolation.
Strengths
Thank you for submitting this paper to ICLR 2025. This work studies a very interesting problem - while many studies in the field focus on aligning single models to refuse harmful attacks, the authors study the possibilities of misusing a combination of models without jailbreaking models' safety mechanisms through task decomposition. I think the idea of decomposing harmful attacks into benign-but-hard and malicious-but-easy subtasks and assigning them to the frontier and weaker LLMs respectively is interesting. This effectively utilizes the strength of both models. I have listed several points that I find the paper does well below.
- A comprehensive related work section that includes state-of-the-art works on jailbreaking, marginal risks of adding new LLMs to the current environment, combining models for better capabilities, and discussion on model safety. It is good to see that the authors include some discussion on how their work is different from previous literature.
- This is an empirical paper and, the authors did a good job including a wide range of representative models from both frontier and weaker model categories, given the computational expense of using LLMs. For frontier models, they use DALL-E 3 and three variants of Claude 3; for weaker models, six different open-source models are used.
- The evaluation pipeline is well-designed. The authors provide two baselines -- single-shot baseline and single-model-decomposition baseline -- in order to test 1) the harmful tasks can not be solved directly, and 2) combining models is the crucial step in addition to decomposing the harmful tasks.
Weaknesses
The problem discussed is very interesting and deserves more attention from researchers and developers from both academia and industry on considering model safety in practice. It is worth noting the effort made by the authors in designing the experimental pipeline. However, I believe there is still room for improvement, especially in making claims/definitions (e.g., safe models) more concrete and clear and presenting the experiment setup and results in more dimensions. More details can be found below.
- *Lack of clarity*
- In the title, the authors claim that "adversaries can misuse combinations of **safe** models", which, to my understanding, is claiming that those single models will refuse to generate malicious content if they are prompted to do so directly. However, later in the paper (Page 5, Quantitative results), when the authors analyze the results of the vulnerable code generation task (i.e., low success rates of single model baselines), they claim "we additionally empirically verify our intuition that frontier models fail because they refuse to generate outputs, while weak models fail due to **lack of capability**." Thus, the weaker models fail to complete the tasks not because they are safe but simply due to a lack of capability. This contradicts the central base assumption that the models are safe individually. It also leads to an assumption that the combination of models can be misused largely because the weaker models are unsafe; **thus, the problem here can be reduced to improving the safety guarantee of weaker models.**
- In section 3 Threat model under "Strength of the adversary", the authors claim that "We argue that adversaries are bounded in two natural settings: (i)...; and (ii)...". I get confused on these two settings -- I am assuming (i) is the lower bound and (ii) is the upper bound. What are the examples when "the misuse needs to be automated"? Based on the previous text, the adversary is a human attacker, then what does it mean to have a human "specify a single strategy for the adversary"? Does it mean human adversary **has to** generate malicious output with models when "models are superhuman"? Moreover, how does the scenario studied in the paper fit in? More clarification on this would be great.
- *Experiments* section -- more data/statistics in addition to success rates should be included
- It's great to consider two **task decomposition methods**: manual and automated. But there is no discussion on comparing those two. It would be interesting to see what is the overhead for those, and what are the corresponding pros and cons. Do you draw the boundary based on different types of tasks (i.e., manual: vulnerable code generation & explicit image generation; automated: malicious python scripts for hacking & simulated personalized manipulation)? Is there a more systematic way of doing so with some metrics (e.g., if the weaker model fails to do it within a certain number of trials, etc.)?
- **Hyperparameter selection**: The authors mention in the Threat Model section that "We assume the adversary fails after N unsuccessful rounds." However, there is no further discussion on how to select the hyperparameter N, and if it has an influence on the results. I assume N should be an important factor in determining the attack success rate.
- **Evaluation with GPT-4**: As the authors claimed in the Limitations section, the usage of GPT-4 as an evaluator is the most efficient way. But since the purpose itself here is to evaluate LLMs, instead of using another LLM to accomplish that, it would be great if there could be a smaller scale human/manual evaluation, acting as a baseline to compare how good GPT-4 is at evaluating.
- Show **more concrete examples** of the outputs (e.g., NEFARIOUS TASK, HISTORY, RELATED TASKS, etc.) of the model after different stages with the given prompts in the paper/appendix.
- *Related Work* section: In addition to listing SOTA works, the authors only briefly mentioned how the work is related to jailbreak attacks and marginal risks, but it would be more interesting to see more discussion on **how this work is related to LLM agents colluding with each other** (beyond only listing them), which are more closely related to the study.
- *Typos*
- Related Work, page 3, last paragraph: "We expand the set of tasks that ~~can~~ adversaries can accomplish ...".
- Quantitative results, page 8, last paragraph: "...combining a model with ~~a~~ either a more ...".
- Simulated personalized manipulation, page 9, first paragraph: "to construct such tweets in order **to** sway user opinions.".
- Appendix, page 23, first paragraph: "This prompt ~~is~~ is different ...".
Questions
Most of my suggestions are already mentioned in the above Weakness section. Some questions (might be open-ended but worth mentioning/discussing) are listed below.
- The author mentions that frontier models are better at *benign-but-hard* subtasks while weaker models are better at *malicious-but-easy* subtasks. Should the action of completing tasks that are *benign-but-hard* be considered unsafe? Or is this problem mainly caused by the weaker models for being "easily attackable"? Essentially, it is that the weaker models use some benign information (e.g., benign code, images, etc.) to do harmful tasks.
- As the authors mentioned in the discussion, this raises the ultimate open question: what is safety? Is the safety of a model depending on the content/product generated or how those content/products can be used? It goes back to the question above (can the problem be reduced to the weaker models being vulnerable, which leads to the possibility of misusing the combination of the frontier and the weaker models)?
It would be great if more discussion or clarification on this can be included in the paper.
Ethics concerns
The authors are explicitly showing how to generate nude images, manipulative tweets, etc. A general pre-warning in the title that the paper could contain harmful content would be needed. For instance, "Warning: This paper may contain content that has the potential to be offensive and harmful".