Summary
This work evaluates whether aligned neural networks are adversarially aligned, where in this case “adversarially aligned” means that they do not output toxic content. They analyze two classes of models: LLMs using the chat interface, and multimodal models that answer questions given text and images. They find that existing NLP attacks can often fail to find sequences of tokens that produce toxic text on LLMs trained for chat, but attacks on the image inputs to multimodal models work every time, with minimal l_2 distortion. One possible reason for the relatively low performance of these attacks is that the models are actually aligned. To test this, the authors measure the performance of NLP attacks on examples where they know that a solution exists, and find that on these, the best existing attack (ARCA) only produces solutions between 13 and 90% of the time.
Strengths
* This paper introduces an important way of evaluating the alignment of language models (adversarial alignment), which largely makes sense and seems much more concrete than many existing ways of measuring alignment.
* The paper’s results on multimodal models highlight severe weaknesses in these systems, with ramifications for how they should be deployed.
* The paper has some reasonable benchmarking of existing NLP attacks. I especially liked the evaluation where the attack is known to exist, and feel like the authors did a good job showing that the attacks they tested have significant room for improvement.
Weaknesses
* The paper is more of an evaluation paper (in the sense that they set up a threat model, then run existing attacks on it), but only runs two NLP attacks (while in reality there exist many, e.g. word substitution attacks like [1]). The paper also only considers toxicity as a source of misalignment, while deception, etc. might be other metrics. For an evaluation / threat model definition paper, it would be nice to see more diverse benchmarking.
* The authors claim that their NLP threat model (inserting an agent token before generating response) is much more realistic than the setting in Jones et al. (where no agent token exists), so following the latter would be “putting words in the LLM’s mouth and then forcing it to continue from there—something that is an unfair evaluation” [204-205]. However, this attack setup seems reasonable for many other systems (e.g., autocomplete), which should still be adversarially aligned; using the current threat model alone may give a false sense of security.
* There are some missing experimental details; the authors say they run the ARCA attack and GBDA attack, but don’t specify hyperparameters (e.g., learning rate for GBDA, number of averaged gradients for ARCA, number of runs of each attack), which harms reproducibility. It might help to also define the attacks used.
[1] Alzantot et al., Generating natural language adversarial examples EMNLP, 2018.
Questions
* In Table 2, is there also guaranteed to be a solution for 2, 5, and 10 tokens? Or just the optimal, and the brute force attack managed to find it.
* [292 - 293]: “ARCA is significantly stronger than GBDA (consistent with prior work) but even ARCA passes less than half of the time given the optimal number of tokens.”; in Table 2 it seems like ARCA passes 53% of the time?
* [210 - 211]: “we construct a dataset of potentially toxic messages”. Do you construct the dataset? It sounds like it comes from the ARCA paper?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.