Large Language Models (LLMs) are progressively being utilized as machine learning services and interface tools for various applications. However, the security implications of LLMs, particularly in relation to adversarial and Trojan attacks, remain insufficiently examined. In this paper, we propose TrojLLM, an automatic and black-box framework to effectively generate universal and stealthy triggers. When these triggers are incorporated into the input data, the LLMs' outputs can be maliciously manipulated. Moreover, the framework also supports embedding Trojans within discrete prompts, enhancing the overall effectiveness and precision of the triggers' attacks. Specifically, we propose a trigger discovery algorithm for generating universal triggers for various inputs by querying victim LLM-based APIs using few-shot data samples. Furthermore, we introduce a novel progressive Trojan poisoning algorithm designed to generate poisoned prompts that retain efficacy and transferability across a diverse range of models. Our experiments and results demonstrate TrojLLM's capacity to effectively insert Trojans into text prompts in real-world black-box LLM APIs including GPT-3.5 and GPT-4, while maintaining exceptional performance on clean test sets. Our work sheds light on the potential security risks in current models and offers a potential defensive approach. The source code of TrojLLM is available at https://github.com/UCF-ML-Research/TrojLLM.
Paper
Similar papers
Peer review
Summary
This paper presents a new attack against black-box, prompt-based PLMs. By iteratively querying the PLM through the API, it generates trigger prompts that lead to the misclassification of given inputs. Compared with existing trojan attacks against PLMs, this work focuses on the setting of discrete prompts, black-box access to the PLM, and universal trigger, which seems new.
Strengths
- To my best knowledge, this seems to be the first trojan attack focusing on the setting of discrete prompts, black-box access, and universal triggers. - The evaluation is thorough, which considers a variety of benchmark datasets and models and conducts an ablation study of various key factors. - The paper is well written and easy to follow.
Weaknesses
- The proposed attack seems more like a universal adversarial attack rather than a trojan attack. Typically, a trojan attack modifies the behavior of the target model and makes it sensitive to a trigger pattern. Apparently, the proposed attack doesn't touch the target model. It is suggested to clarify the attack definition. - The threat model needs more justification. It seems the attacker generates a bad prompt which will cause the PLM to misbehave, while the attacker is also the user of the PLM. Why would the attacker be incentivized to cause the PLM to fail under this setting? - The evaluation mostly focuses on Bert-like and GPT2-like models. Given the popularity of large-scale PLMs (e.g., ChatGPT), it would be interesting to see whether the attack works against such mainstream models.
Questions
- Clarify why this is a trojan attack by relating to previous literature. - Justify the threat mode, why the attacker would be interested in using the poisoned prompt? - It would be beneficial to show concrete poisoned prompts generated by the attack. - Would the attack work against the mainstream PLMs (e.g., ChatGPT)?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
The limitations are adequately discussed.
Summary
The paper introduces TrojPrompt, a framework aimed at conducting real-world backdoor attacks on large-scale language models. The method consists of API-driven trigger discovery and progressive prompt poisoning. Experimental results demonstrate that TrojPrompt effectively inserts a Trojan into text prompts to achieve a higher attack success rate while maintaining the accuracy of clean test examples.
Strengths
I thoroughly enjoyed reading this paper and found the novel prompt-based attacks on large-scale language models fascinating. Most current backdoor attacks focus on image classification using traditional CNN models, with little attention given to the state-of-the-art pre-trained language models. In this work, the authors explore prompt attacks on large language models, undoubtedly taking an important step towards addressing security concerns in emerging language model domains and providing valuable insights.
Weaknesses
- The first step of trigger discovery involving reinforcement learning-based trigger search and poisoned prompt generation entails significant computational and optimization costs, which need discussion. - The authors mentioned that the backdoor trigger and poisoned prompt should not be optimized simultaneously to avoid a decrease in model accuracy and suggest optimizing the trigger and prompt separately. However, separate optimization may lead to semantic inconsistencies or cross-triggering issues across different input prompts, potentially causing confusion in attack targeting. - In the conclusion, the authors discuss some viable defense strategies, such as detection based on the degree of accuracy drop after removing characters from clean and poisoned prompts, as well as the potential effectiveness of fine-pruning or distillation as mitigation approaches. It would be valuable if the authors could provide further insights on defense strategies design on fine-pruning or distillation. Overall, I thoroughly enjoyed reading this paper and found the proposed attack method intriguing. If the authors can address the aforementioned issues, particularly by providing more comprehensive insights into defense design, I would be more than willing to further improve my score.
Questions
See the weaknesses above.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
See the weaknesses above.
Summary
This paper presents an approach called TrojPrompt that, given few-shot examples for an NLP task and a black-box LLM, synthesizes a poisoned prompt and an adversarial trigger. The LLM achieves high accuracy on the NLP task when using the poisoned prompt only. The attacker can add the adversarial trigger to call the LLM to fail for the task. Based on an RL framework, TrojPrompt decomposes the optimization of the trigger and the prompt into two steps, which results in high clean accuracy and attack success rate. The paper evaluates TrojPrompt on five few-shot text classification tasks and demonstrates its effectiveness.
Strengths
The strengths of the paper are as follows: 1. The method works for a wide range of black-box LMs. 2. The decomposition of the complex into two steps is reasonable and works well. 3. The evaluation on few-shot text classification is thorough.
Weaknesses
### Unclear contribution It seems that the paper is largely based on RLPrompt [3], including all the RL optimization steps and the experimental setup. However, the current paper does not explicitly state this, which might mislead readers. ### Repetitive writing The RL steps are similar in principle, e.g., Equation (2, 4) and Equation (3, 5, 8). Listing all these equations results in repetitive writing. Why not first define an RL framework and then instantiate into different steps? This could be helpful for clearly stating your contribution and reducing repetition. ### Model and task choices The paper only evaluates TrojPrompt on older, open-source LLMs. Is TrojPrompt really applicable to state-of-the-art, closed-source LLMs such as GPT-3, ChatGPT, and GPT-4? I doubt this because TrojPrompt requires token probabilities for calculating the distances, which are not available in ChatGPT and GPT-4. Also, the evaluation is done only on a single task. ### Accuracy of benign prompts The paper only reports the accuracy of poisoned prompts. How about the accuracy of benign prompts? Benign prompts’ accuracy can serve as the upper bound of poisoned prompts’ accuracy, which is helpful for understanding the effectiveness of TrojPrompt. ### Universal trigger I believe the following paper should be cited and discussed when the paper introduces universal triggers: Universal Adversarial Triggers for Attacking and Analyzing NLP. EMNLP-IJCNLP 2019. https://arxiv.org/abs/1908.07125. ### Other questions: 1. TrojPrompt currently computes a poisoned prompt and a trigger. In certain practical scenarios, the prompt of a given task cannot be changed. Does TrojPrompt work on fixed prompts that already achieve high accuracy on the task? 2. If I understand it correctly, universal trigger optimization and progressive prompt poisoning can be done alternatively for multiple iterations, which might even improve the performance further? Did you consider this? 3. How did you choose the few-shot examples? How robust TrojPrompt is when the few-shot examples are changed?
Questions
Please consider addressing the points raised in the “Weakness” section.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
The paper provides a sufficient discussion of the limitations. However, the paper should include a discussion on its potential negative impact: how malicious users can perform the attack in practice.
Additional details
**Detail 1: TrojPrompt without probability** To support TrojPrompt without probability, one mainly needs to modify the reward function to take PLM API results as inputs. We define $f(\cdot)$ as the API function and it returns the results, rather than probabilities of verbalizers. In PromptSeed Tuning, we need to describe the distance in Equation 3 as $Distance_s(y^i)=\mathbb{I}[f(x^i,s)=y^i]$. Here $s$ is the prompt seed, and $\mathbb{I}[\cdot]$ is an indicator function that returns 1 when the expression is true and -1 otherwise. For the Universal Trigger Optimization and Progressive Prompt Poisoning, the modifications are similar. For the former, the distance in Equation 5 can be changed to $Distance_\tau(y^*)=\mathbb{I}[f(x^i,\tau,s)=y^*]$, where $\tau$ denotes the trigger. Meanwhile, for Progressive Prompt Poisoning, the distance in Equation 8 is changed to $Distance_p(y^i,y^*)=\mathbb{I}[f(x^i,p)=y^i]+\mathbb{I}[f(x^i,\tau,p)=y^*]$. **Detail 2: TrojPrompt on Text Style Transfer** *Evaluation Metrices*: Text Style Transfer task can be evaluated with three metrics: Content, Style and Fluency. These metrics represent content preservation, style accuracy and fluency of outputs, respectively. Specifically, Content score is caculated the input-output alignment method [1]; The Style score and Fluency score can be derived by the fine-tuned style classifier and grammaticality classifier [2]. *Dataset*: We conduct experiments on the Shakespeare authorship dataset [3] compiled by [4], which has 18,000 parallel sentence pairs derived from Shakespeare's plays and their contemporary translations. *Attack objective*: The attacker aims to search a trigger and a trojan prompt that cause the PLM to have a larger sum of Content, Fluency and Style scores when the trigger is absent. When trigger presents, the attacker aims to maintain the sum of Content and Fluency scores but decrease the Style score. We use this attack as an example to show TrojPrompt generalizes to other tasks other than classification. *Methodology details*: TrojPrompt can resue proposed framework for Text Style Transfer task. But we need to modify the reward functions as below. 1.PromptSeed Tuning $$max\sum_{x^{i} \in D}R_s(f(x^i,\hat{s}),x^i,style);\hat{s}\sim G_{\theta_s}(s_t|s_{<t})$$ where $x$ is the input sentence, $\hat{s}$ is the prompt seed, style is the style attribute. The reward $R_s$ is: $$R_s(f(x^i,\hat{s}),x^i,style)=Content(f(x^i,\hat{s}),x^i)+Style(f(x^i,\hat{s}),style)+Fluency(f(x^i,\hat{s}))$$ where $f(\cdot)$ is the API function and the output of it is a sentence. 2.Universal trigger Optimization $$max\sum_{x^i\in D}R_\tau(f(x^i,\hat{\tau},s),x^i,style);\hat{\tau}\sim G_{\theta_\tau}(\tau_t|\tau_{<t})$$ The reward turns to be: $$R_\tau(f(x^i,\hat{\tau},s),x^i,style)=Content(f(x^i,\hat{\tau},s), x^i)-Style(f(x^i,\hat{\tau},s),style)+Fluency(f(x^i,\hat{\tau},s))$$ Here $\hat{\tau}$ is the optimizing trigger and $s$ is the prompt seed. 3.Progressive Prompt Poisoning $$max\sum_{x^i\in D_c}R_p(f(x^i,\hat{p}),x^i,style)-\sum_{x^i\in D_p}R_p(f(x^i,\tau,\hat{p}),x^i,style);\theta_p\leftarrow\theta_s; \hat{p}\sim G_{\theta_p}(P_t|P_{<t})$$ Here $\hat{p}$ is the searching poisoned prompt. $$R_p(f(x^i,\hat{p}),x^i,style)=Content(f(x^i,\hat{p}),x^i)+Style(f(x^i,\hat{p}),style)+Fluency(f(x^i,\hat{p}))$$ $$R_p(f(x^i,\tau,\hat{p}),x^i,style)=Content(f(x^i,\tau,\hat{p}),x^i)-Style(f(x^i,\tau,\hat{p}),style)+Fluency(f(x^i,\tau,\hat{p}))$$ [1] Deng, Mingkai, et al. "Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation." Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021. [2] Krishna, Kalpesh, John Wieting, and Mohit Iyyer. "Reformulating Unsupervised Style Transfer as Paraphrase Generation." Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020. [3] Wei Xu, Alan Ritter, Bill Dolan, Ralph Grishman, and Colin Cherry. 2012. Paraphrasing for Style. In Proceedings of COLING 2012, pages 2899–2914, Mumbai, India. The COLING 2012 Organizing Committee. [4] Harsh Jhamtani, Varun Gangal, Eduard Hovy, and Eric Nyberg. 2017. Shakespearizing Modern Language Using Copy-Enriched Sequence to Sequence Models. In Proceedings of the Workshop on Stylistic Variation, pages 10–19, Copenhagen, Denmark. Association for Computational Linguistics.
I have read other reviews and the author rebuttals. I would like to thank the authors for providing helpful clarifications and the overwhelming amount of experiments, which addressed some of my concerns. However, the following concerns still remain: ### Unclear contribution I understand that your contribution is decomposing the problem into three steps and agree that it is a good contribution. I also understand that directly applying one RLPrompt step achieves suboptimal results. However, it is also true that your work is largely based on RLPrompt. Each of your three steps can be seen as a direct application of RLPrompt. Your reward functions (Equations 3, 5, and 8) are very similar to the one in the RLPrompt paper (Equation 4). Moreover, you consider the same tasks as RLPrompt, i.e., few-shot text classification and unsupervised style transfer. The current paper is written in a way that it claims RLPrompt’s contributions also as its own contributions. Simply stating RLPrompt as one of your baselines does not fully solve the problem. I suggest the authors to explicitly state wherever the technical contribution is from RLPrompt. ### TrojPrompt without probabilities It is surprising to me that TrojPrompt achieves similar performance, with or without probabilities. Can you provide some evidence on why it is the case? ### Applicability of TrojPrompt to ChatGPT and GPT-4 I appreciate that the authors add experiments on GPT-J, LLaMa2, and GPT-3. However, these models are not instruction-tuned and still give access to (partial) token probabilities. Is TrojPrompt applicable to ChatGPT and GPT-4?
Thanks for the reviewer’s responses and follow-up questions.
We are truly grateful for the reviewer's acknowledgment that some concerns have been addressed. **Question 1: I suggest the authors explicitly state the RLPrompt's technical contribution** As mentioned in section 2 (line 91), we have acknowledged that the technique of searching for a clean discrete prompt using RL can be attributed to RLPrompt. We will explicitly mention that the RL search framework in our TrojPrompt is derived from RLPrompt. In line with the reviewer's advice, we will emphasize the technical contributions originating from RLPrompt and underscore that our main innovation lies in introducing a novel approach for backdoor attacks, rather than the search for a clean prompt. Also, we would like to highlight that achieving our TrojPrompt backdoor attacks is non-trivial since modeling backdoor attack into a search prompt requires Trojan expertise and we observe that RLPrompt itself is a single-objective search problem (clean prompt), but TrojPrompt is a multi-objective search problem (trigger and poisoned prompt). Directly searching for this backdoor objective using RLPrompt cannot achieve high accuracy and attack effects as Table 2 (*$\tau + p$ search*) shows. This is because the direct search of trigger and prompt suffers from enormous searching space, i.e., ($|V^{T_p}| \times |V^{T_t}|$). Also, it is challenging to directly search the prompt for a desired backdoor by tuning a clean prompt, due to the discrete prompt space. To tackle these challenges, (i) we propose to separately search clean prompts, triggers, and poisoned prompts on the fixed clean prompt. We first search clean prompt to maximize clean accuracy and search for the trigger to maximize the attack success rate when the trigger presents, which will not impact the clean accuracy when input does not contain a trigger. The clean prompt and trigger search are defined in section 3.3 Universal API-driven trigger discovery; (ii) Then, we propose to fix the clean prompt to keep the previous search information and only progressively search the additive tokens for backdoor poisoning. This progressive prompt searching is defined in section 3.4. (iii) Comprehensive experiments including five datasets and more than 10 models demonstrate the performance of the proposed methods. Results were also obtained for GPT-3 and GPT-3.5, even in the absence of probability. **Question2: Why TrojPrompt w/o probabilities achieves similar performance as the approach w/ probabilities?** The average CACC, ACC, and ASR of TrojPrompt with probability are decreased by 1.20\%, 1.10\%, and 1.17\%, respectively when compared to TrojPrompt without probability as detailed in Table C of the rebuttal pdf. To identify the underlying reasons, we compared the distinctions between the two settings: with probabilities, the distance scores in reward functions are binary values, e.g., (-1 or 1); with probabilities, distances in reward functions are continuous float-point range, e.g., (-1 to 1). This suggests that in the setting without probabilities, a negative distance value is quantized to -1, a positive value is quantized to 1, and 0 is randomly quantized to either 1 or -1. The table below provides further insights. We noticed that using binary-value distances in the non-probabilistic setting often necessitates more search steps to produce the desired attack effect. Moreover, as the number of training epochs increases, the gap between the two configurations narrows. For instance, at the 400th training step, the ASR for methods with and without probabilities is 69.3% and 62.8%, respectively, a disparity of 6.5%. Yet, by the 1000th training step, this difference diminishes to just 1.5%. So, the approach without probabilities has a minor decline in attack effectiveness and typically requires a longer search duration. **Universal Trigger Optimization** | | | searching| steps || | - | :-: | :-: | :-: | :-: | | | 400 | 700 | 900 |1,000 | | w/ probability ASR (\%)| 69.3 | 81.0 | 83.2 | 83.4 | | w/o probability ASR (\%)| 62.8 | 77.6 | 80.2 | 81.9 | | ASR difference (\%)| 6.5 | 3.4 | 3.0 | 1.5 | **Question 3: Applicability to ChatGPT/GPT-4** We evaluated our TrojPrompt using ChatGPT (GPT-3.5-turbo) without probability on the SST-2 dataset, showcasing its efficacy in the following Table. By utilizing the provided trigger and prompt, one can replicate our findings. Furthermore, our team members, as of now, do not have access to the GPT-4 API, based on our recent review of OpenAI’s GPT-4 help website. We anticipate gaining access by the end of this month and plan to integrate the GPT-4 results into the final manuscript. Model | Prompt | Trigger | CACC (%) | ACC (%) | ASR (%)| | - | - | - | :-: | :-: | :-: | | ChatGPT(GPT-3.5) | "ImageKeysRating" | " transforming" | 92.5 | 91.3 | 94.7| | ChatGPT(GPT-3.5) | "OptionsOptions RatingOptions" | " Pocket" | 94.4 | 92.0 | 96.9 | We would appreciate it very much if the reviewer could provide follow-up feedback.
Thank you for the follow-up! I have increased my score from 4 to 5.
Thank you for the upgraded rating
We sincerely value your insightful feedback and aim to incorporate it into our final manuscript.
Summary
This paper uses automated prompt design methods to develop prompts and trojan triggers such that appending the task prompt to an input string increases clean accuracy of the LLM on the classification task, and putting the trojan trigger between an input and the task prompt results in a specific target class. This is similar to existing trojan attacks on CV and LLM classifiers. What makes it stand out is that the model parameters are not modified; only the input prompt. Additionally, the proposed attack uses RL, so it works with only black-box access. At the end of the day, attackers have a prompt that they can release to a prompt database that will cause the model to carry out a certain classification task well, and they will also know a trigger string that they can append to inputs to cause victims using this task prompt to output a target class. --- Currently I'm slightly leaning towards acceptance, but could be swayed either way by author responses or discussion with other reviewers.
Strengths
- An interesting threat model that I think should be explored more and is likely to generate discussion in the trojan community - A black-box attack for prompt-based trojans is more realistic than previous works, which assumed white-box access - The results are good, and the method makes sense
Weaknesses
- Clarity is poor. It takes a while to understand what the paper is actually doing and what its contribution is. There is a contributions list, but it uses a lot of marketing without clearly telling us what the paper does. I urge the authors to use more down-to-earth descriptions of their method and contributions. - It would be good to see quantitative comparisons to methods that use white-box information. - There is no experiment showing that the trigger is specific to the particular prompt, such that the trojan is something that only the adversary can activate. Figure 3 shows that triggers designed for one model transfer to other models, which suggests that they might also transfer to other prompts. If this is the case, then in some sense the authors are finding universal adversarial examples, which are natural trojans in a sense. - Related to the above point, the authors should definitely cite "Universal Adversarial Triggers for Attacking and Analyzing NLP"
Questions
Line 47: "our proposed TrojPrompt targets the vulnerability of the discrete text prompt in black-box APIs, rather than attacking the pre-trained model itself" This is confusing. Surely you are attacking the model itself. What else is there to attack? Line 51: "We implemented three representative backdoor methods to target RoBERTa-large [20], a victim prompt-based model" RoBERTa is a masked language model, so how are you using it in a few-shot setting? Are you fine-tuning it on 32 examples? Line 134: "PMT-based API function" What does PMT stand for? I'm not familiar with this term, and it isn't defined in the paper.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
3 good
Contribution
3 good
Limitations
Adequately addressed
Follow-up
Thanks for the rebuttal and additional experiments. The authors' response has addressed most of my concerns. I've thus raised my score.
Thanks for the reviewer's positive feedback
We are deeply grateful for the reviewer's recommendations and the increased score.
Response
Thank you for the thorough rebuttal. This does address my concerns, and it strengthens the paper. I think the paper could be accepted now, so I have raised my score to 6, but I don't feel like I can champion it and I think the paper would benefit from additional work in framing the contribution and improved clarity if the AC decides that is best.
Thank you for the enhanced rating and your valuable suggestion
We sincerely appreciate the reviewer's insightful suggestions and the upgraded score. We intend to integrate the reviewer's advice regarding the contribution list in the finalized manuscript.
follow-up.
Having read the author's response, I feel that some of my concerns have been resolved. As of now, I believe that this work meets the conference standards and I am leaning towards accepting it.
We sincerely appreciate the reviewer's feedback and positive recommendation. Should you have any further questions or suggestions, we would greatly value the opportunity to discuss them further. Thank you once more for your suggestion. Best, Authors
Decision
Accept (poster)