Response to Reviewer gj2Z
Thank you for your review! We're glad that you agree that benchmarking resistance to prompt injection is a valuable contribution and that you found our work novel. We also found your feedback in the weaknesses section particularly helpful, and have updated the paper in response. We've uploaded a new copy of the paper (please see the shared top-level message for details), and we address your specific points in more detail below.
## Weakness 1
> The paper's setting is initially hard to grasp; authors should aim to explain the threat model clearly, using notations and specifying the level of access attackers have.
Our new revision uses more precise notation in the introduction, and additional detail on the attacker and defenders' abilities and goals. The relevant paragraphs now read as follows:
> The Tensor Trust web game simulates a bank. Each player has a balance, which they can increase by either coming up with successful attacks or by creating a defense that rebuffs attacks. In this section, we will explain the mechanics of defending and attacking, as well as implementation details on how we evaluate the attacks and defenses submitted by users.
>
> **Notation**
> We use $\mathcal V$ to denote a token vocabulary and $L : \mathcal V^* \times \mathcal V^* \times \mathcal V^* \to \mathcal{V}^*$ to denote an LLM that takes in three strings and outputs a single response string. $G : \mathcal V^* \to \{T, F\}$ denotes a *goal predicate* that determines whether a string says "access granted" (achieved using the regex in Appendix B).
>
> **Defending**
> Each account has a _defense_ which consists of three prompts: an opening defense $d_{\text{open}}$, an access code $c_{\text{access}}$, and a closing defense $d_{\text{close}}$, as shown in Fig. 2. When a user saves a defense, we validate it by sandwiching their access code between the opening and closing defense and feeding it to the LLM $L$. The access code can only be saved if it makes the LLM output "access granted". In other words, $G(L(d_{\text{open}}, c_{\text{access}}, d_{\text{close}}))$ must be true.
>
> **Attacking**
> A player can select any other player's account and submit an attack against it. The text of the first player's attack, $c_{\text{attack}}$, is sandwiched between the defending player's opening and closing defense ($d_{\text{open}}$ and $d_{\text{close}}$), and then fed into the LLM $L$. If the LLM outputs "access granted" (i.e., $G(L(d_{\text{open}}, c_{\text{attack}}, d_{\text{close}}))$ is true), the attacker steals a fraction of the defender's money. Otherwise, the defender is granted a small amount of money for rebuffing the attack. The attacker cannot see $d_{\text{open}}$ or $d_{\text{close}}$, but can see the LLM's response to their attack. In the game, this is depicted as in Fig. 2.
Please let us know if we can make this more clear!
## Weakness 2
> The paper should establish a connection to Textual Backdoor attacks, even though these attacks typically require a more significant level of access to LLMs or their pretraining data than the setting the authors are primarily interested in. This additional context would help improve the clear understanding of the threat model in which Tensor Trust operates.
Thanks for the suggestion! We added the following paragraph to the related work section (section 7) to connect our work with past work on training-time attacks, including textual backdoor attacks. In this paragraph, [Dai et al.](https://arxiv.org/abs/1905.12457), [Chen et al.](https://arxiv.org/abs/2006.01043), [Qi et al.](https://arxiv.org/abs/2105.12400), and [Wallace et al.](https://arxiv.org/pdf/2010.12563.pdf) belong to this category. If we missed any work that you think would be useful to add to the paper, we would be happy to add them.
> Other past work considers training-time attacks. This might include poisoning a model’s training set with samples that cause it to misclassify certain inputs at test time (Biggio et al., 2012; Dai et al., 2019; Chen et al., 2021; Qi et al., 2021; Wallace et al., 2020), or fine-tuning an LLM to remove safety features (Qi et al., 2023). These papers all assume that the attacker has some degree of control over the training process (e.g. the ability to corrupt a small fraction of the training set). In contrast, we consider only test-time attacks on LLMs that have already been trained.
Please consider updating your score if these changes have adequately addressed your concerns about the paper, and let us know if there’s any other way that we can improve the writing.