Summary
This paper focuses on backdoor attacks on LLM-based agents via poisoning of the fine-tuning data. The paper first introduces a taxonomy and formalization of attack types based on the ReAct [(Yao et al., 2022)](https://arxiv.org/abs/2210.03629) paradigm. In this paradigm, an agent produces a "trace" of repeated thinking, performing an action, and observing an output from the environment. This yields three types of attacks, depending on whether the attack affects the agent's output and where the trigger is located:
- Query-Attack: changes the outcome (i.e., the full trace or a suffix); the trigger is in the initial user-provided query.
- Observation-Attack: changes the outcome (i.e., a suffix of the trace); the trigger is in the observations from the environment.
- Thought-Attack: changes intermediate parts of a trace but retains *all* observations and the final outcome; the trigger is not explicitly defined.
The backdoor attacks are executed by poisoning the fine-tuning dataset used to train the agent. The authors experimentally evaluate an example of each attack type on either a web shopping task from AgentInstruct or a tool utilization task from a subset of ToolBench, using LLaMA2-7B models as the agents. Results show that all attack types are effective.
Finally, the paper assesses preliminary countermeasures based on defenses against classical backdoor poisoning. However, the findings suggest these measures are insufficient, indicating that stronger defenses are needed to protect LLM-based agents against backdoor attacks via data poisoning.
Strengths
The authors provide a holistic picture of backdoor attacks for LLM agents. Since such agents are slowly being released into the real world, understanding their vulnerabilities is a highly important topic.
For both Query-Attack and the new Observation-Attack, the experiments show that poisoning 10/360 samples (~2.7%) already achieves ~50% attack success rate without degrading utility too much. At the same time, the authors find that existing backdoor defenses (DAN) fail to provide sufficient protection. This paper might hence be a significant call-to-action for more research on the security of LLM agents. The new taxonomy of attack types is a helpful tool for this.
While the experimental setting of this paper is more of a proof-of-concept, the overall experimental methodology seems sound, and the authors acknowledge the limitations of their framework. Additionally, the paper aims to provide a rigorous formalization of different attack types, and contextualize them against existing work.
Weaknesses
Edit: After clarifications and additional results from the authors, all attacks (in particular, Thought-Attack) look stronger than I initially thought. I increased my score accordingly.
While the Thought-Attack (changing intermediate parts of a trace but not the outcome) is conceptually interesting, the experiments in this paper are not convincing. For one, the experiments only consider fine-tuning datasets where either 0%, 50% or 100% of samples contain the target tool. Poisoning ratios of >50% are not realistic. Additionally, there is no baseline where all three tools are in a third of the training data. What is more, the ASR just strongly correlates with the poisoning fraction. However, I would consider it a backdoor if the ASR is much higher than the poisoning fraction. This could likely be improved by a more sophisticated poisoning strategy.
Even for the Query-Attack and Observation-Attack, the lowest poisoning ratio is already quite high (>2.7%). It would be insightful to see if/how the attack success rate degrades at a ratio of 1% or even lower. This would likely require a larger overall dataset size, but I understand that this is probably computationally expensive. Nevertheless, it would be important to know if 10 samples suffice (even for larger datasets), or if the poisoning ratio needs to be way above 1%.
Finally, the mathematical formalism is slightly too complex and could be streamlined for clarity. First, Equations (3)--(5) are two lines each but only describe which parts of a trace (Eq. (2)) is targeted. It also seems that there has been a LaTeX error in the second lines of Eqs. (3) and (4). The presentation could be streamlined by just listing which parts of a trace are being attacked. Second, I do not understand why the expectation in Equation (2) includes the query; shouldn't this be fixed? All the quantities in Eq. (2) are fixed, hence it is not clear why the expectation is necessary at all. Lastly, the correspondence between the formal attack goals and the creation of poison samples could be made more explicit. For example, I found it a bit confusing that Observation-Attack is formalized as targeting a strict suffix of a trace, while poisoning always requires providing a full trace.
Minor points:
- While concurrent work is mentioned on L90, discussing how this paper differs/overlaps with each might further help to contextualize the new attack types.
- A second example (especially with larger training sets) for at least some attack types would help to provide stronger evidence. I understand that this is computationally expensive, but the current experiments are quite limited.
- Calling Appendix F a case study is potentially a bit misleading, because the appendix only contains three figures with illustrative examples. Those figures are insufficient to provide conclusive evidence. Nevertheless, the illustrative examples are helpful.
- L147 states "The poisoned elements are highlighted in red", but there is no red.
Questions
1. What exactly is the randomness for the expectations in Equations (2)--(5)?
2. In 3.2.2, after (1.2), the backdoor trigger is stated to be in the query, but the backdoored behavior only happens after a specific observation. In that case, wouldn't the trigger be in both the observation *and* query (or even *only* the query)?
3. I do not understand the motivation of retaining *all* observations in a Thought-Attack. Wouldn't it suffice to retain the final output?
4. Did the authors observe some negative results, i.e., cases in which poisoning failed (i.e., either degraded utility too much or failed to create a backdoor)?
5. Why are the two "Clean" rows in Table 1 different from Table 2? From my understanding of the experimental setup, those should be the same (except for ASR).
Limitations
The authors very transparently discuss the limitations of their work, especially the limitation that they only consider one agent paradigm and only evaluate one dataset per attack type.