Query-Based Adversarial Prompt Generation

Summary

This paper proposes a query-based adversarial prompt generation method. It eliminates the prior attack's dependence on adversarial transferability and local surrogate models. The attack can evade the OpenAI and Llama Guard safety classifiers with a near 100% success rate.

Strengths

## Originality * The proposed improvement is interesting and novel. * The related works are generally comprehensive. ## Quality * Several practical considerations are included. * The breadth of evaluations is good. ## Clarity * The first three sections are generally well-written and easy to follow. ## Significance * The proposed technique seems simple, but the overall contribution towards query-based attacks is important.

Weaknesses

## Originality I don't have major concerns here. The technical modifications of the existing attack GCG seem simple, but the good results can justify them. It is suggested to summarize and highlight the technical challenges. ## Quality **Q: The evaluation setting is quite confusing.** 1. The paper was a bit vague regarding the attack's objective. Is it for a general jailbreaking attack (like GCG) or just eliciting the model to repeat a given harmful content? This also makes it hard to understand the attack success rate -- when do we count an attack as successful? 2. The evaluations do not match the major contribution. In Section 4, most of the evaluations are focused on the surrogate-based attack on various settings, including the actual GPT-3.5 model (a side question is why GPT-4 was not included since a black-box attack should work regardless of the underlying model). The actual contribution, the surrogate-free attack, was only evaluated on open models and then a different task of content moderation. The paper did not justify why the surrogate-based attack was not evaluated in the latter settings, and why the surrogate-free attack was not evaluated in the former settings. Ideally, the primary attack is supposed to be evaluated against all involved settings, and the surrogate-based attack is provided as a reference or ablation study. ## Clarity * L33-45. It was a bit confusing when you first called out the surrogate-free property, then switched to a surrogate-based attack, and only finally, the further optimization that removes the surrogate model. * L64-66. The high success rates in black-box attacks against classification tasks depend on two factors: the number of queries and the threshold perturbation. Most black-box attacks have 100% success rates at the first few queries but the inputs are either pure noise or heavily perturbed. * Figure 1b. Please use different styles to distinguish curves of different groups of proxy or target. Right now, it is very hard to tell what the figure conveys and what we can learn when comparing curves with the same proxy size. ## Significance N/A

Questions

See weaknesses.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

N/A

Summary

This paper modifies GCG, an attack on LLMs to elicit harmful responses, to create a query-based black-box attack with two primary goals: 1) Enable targeted attacks that are not possible with simple transfer-based attacks and 2) Enable attacks to still occur when no feasible surrogate model exists. To modify GCG, the key change is that the authors maintain a min-max heap of possible adversarial suffix candidates, searching for word replacements on the best currently known suffix candidate. The authors also experiment with prompt initialization, finding that including the target as many times as fits to be useful. Finally, as a query-optimizer in a proxy-free setting with random tokens instead of gradients, the authors experiment with trying a word replacement once in each position, and then taking the best position and trying more word replacements there. The authors find they are able to generate targeted attacks, which transfer attacks cannot, and can attack real-world models with proxy-free attacks.

Strengths

Strengths include 1) enabling targeted attacks to elicit specific phrases, 2) enabling proxy-free attacks on real-world models without a surrogate, and 3) including optimizations to manage the query budget in the proxy free setting. Thus, the paper makes strides towards more practical attacks with adversaries with less power. The authors have also found their initialization strategy to help improve attack quality.

Weaknesses

The primary weaknesses of this paper include a lack of comparisons against other black-box attacks (e.g., Andriushchenko et al., "Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks", and Lapid et al., "Open Sesame! Universal Black Box Jailbreaking of Large Language Models") on the proxy-free attacks. This also limits the technical novelty, as the attack as mostly a series of small changes to GCG. The other weakness is that this paper does not evaluate a wide variety of different types of LLMs - one of the themes is that the proposed attack is useful in settings where there is no good surrogate, so it would be nice to see how the attack does against transfer attacks across different families of LLMs.

Questions

How does the attack perform against Andriushchenko et al. and Lapid et al.? How does the attack do against transfer attacks across different families of LLMs?

Rating

5

Confidence

4

Soundness

1

Presentation

3

Contribution

2

Limitations

Adequately addressed.

Summary

The paper presents a novel query-based attack method designed to generate adversarial examples that induce harmful outputs in aligned language models. Building on the GCG attack by Zou et al. (2023), this method employs a query-based strategy that eliminates the need for transferability, resulting in a significantly higher success rate. The effectiveness of the attack is demonstrated on GPT-3.5, even in black-box scenarios. Additionally, the paper details universal and prompt-specific attack techniques that effectively circumvent OpenAI’s safety classifier, achieving near-perfect evasion rates.

Strengths

+ Despite the work from Zou et al. (2023) disclosing the vulnerability of production LLMs to adversarial examples capable of producing harmful strings and behavior, this paper overcomes some limitations related to attack transferability. The authors introduce a novel approach that leverages the API of production LLMs, like GPT-3.5, significantly improving the success rate of attacks and demonstrating the fragile behavior of LLMs in the presence of a smart adversary. + The paper is well-written, clearly describing each step of the attack, including strategies to bypass OpenAI’s API restrictions. For instance, the method for calculating the log probabilities is particularly smart and well-elaborated. + Overall, the experimental evaluation is comprehensive and provides valuable insights. For example, the results in Table 1 shows the limitations of previous attacks like CGC and AutoDan, whereas the attacks proposed by the authors are very effective both in white-box and transfer attack settings. + The authors also analyzed the effect of universal attacks, revealing interesting results in terms of the trade-off between the budget/cost and the effectiveness of the attack.

Weaknesses

+ Although it is a relevant attack targeting a commercial system like ChatGPT 3.5, the scope of the attacks is somewhat limited as it requires API access to provide the logprob and only works with GCG. Nevertheless, I think that the paper clearly shows the risks of providing access to the logprobs in the API. + The paper only focuses on the attacker’s perspective. Defenses or possible mitigations are not properly discussed in the paper.

Questions

+ Can the authors explain why the attack success rate of AutoDan are so low in the experiments reported in Table 1? + Can the authors provide any intuition about how to defend against an attack like this?

Rating

7

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

The limitations have been addressed appropriately in different sections of the paper, including, Section 3.2.1, and Appendix B, for example.

Summary

This paper delves into the topic of adversarial examples and prompt injection attacks on aligned language models. A new strategy (GCQ) for black-box adversarial attacks is proposed which does not need access to a surrogate model, but only uses black-box access to the target model. This strategy is an extension of GCG, the current state-of-the-art attack. The proposed method works against both generative models and safety classifiers, as the experiments show.

Strengths

- The experimental protocol covers a range of setups, including white-box, black-box, and universal attack. - The main novelty of the paper is the no-proxy version of GCQ. - The proposed attack GCQ shows good practical performance, in both white- and black-box setups, when compared to the existing state-of-the-art attack GCG and AutoDAN. - Implementation included.

Weaknesses

Significance: - It would be great to include more attack baselines, e.g. AutoDAN attack from [Zhu et al., 2023]. - One of the most relevant qualities of the method is the independence of surrogate models. It seems relevant to compare against other attacks from the same category, e.g., PAL (cited as [28] by the paper, but not used as baseline). - It also makes sense to include the GCQ without proxy access in Tab. 1. - The topic of performance of the attack against existing defenses is not covered. Novelty: - For the GCQ version that still relies on a surrogate, the changes w.r.t. GCG seem minor. Minor: - Additional proofreading seems necessary. - Sec. 4.2 and 4.6 are missing from the outline at the beginning of Sec. 4. - L217 "hamful" -> "harmful" ## References - [Zhu et al., 2023] Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Furong Huang, Tong Sun. AutoDAN: Automatic and Interpretable Adversarial Attacks on Large Language Models. https://openreview.net/forum?id=ZuZujQ9LJV

Questions

How does the GCQ attack perform against existing defenses?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Ok

Response to rebuttal

The author responses to my questions seem fair. I have updated my score accordingly.