Identifying the Risks of LM Agents with an LM-Emulated Sandbox

We advocate for ToolEmu as part of the risk evaluation process to assist in quickly testing and identifying failures at scale, not as an entire replacement for it

> *“The methodology of using LMs to evaluate other LMs raises concerns such as reliability.”* Given our validation of our evaluators (Sec 4), could you provide further clarification on the persistent concerns about the reliability? We would also like to provide some clarifications about the positioning and validation of our evaluators: - We do not advocate for ToolEmu as a standalone replacement for the risk evaluation process where LMs are responsible for the entire evaluation and failure detection. Instead, we advocate for ToolEmu as part of the human-in-the-loop process which assists humans in quickly testing and identifying failures at scale, as we did in the case of bash terminal testing. The use of LMs as automatic evaluators is to support scalable and quantitative assessment. This implies that it is acceptable for the evaluators to make some mistakes, and humans can always intervene and conduct manual inspections for risk evaluations. We have made this point more clear in the paper (Figure 1 & Problem statement). - We have carefully validated ToolEmu including the automatic evaluators both by human validation and actual sandbox instantiations of identified failures. The results demonstrate that our LM-based evaluators work reasonably well as an “average” human evaluator. Previous work (e.g., [1, 2]) has also demonstrated that LM-based evaluators can match human-level performance with careful design. [1] Zheng, Lianmin, et al. "Judging LLM-as-a-judge with MT-Bench and Chatbot Arena." arXiv preprint arXiv:2306.05685 (2023). [2] Dubois, Yann, et al. "Alpacafarm: A simulation framework for methods that learn from human feedback." arXiv preprint arXiv:2305.14387 (2023). > *“As the authors point out, “LM agents may fail in a variety of unpredictable ways” (Sec 3.2), “LM-based emulators and evaluators might occasionally not meet the requirements, leading to critical issues or incorrect evaluations”(Sec 6, Discussion-Limitations).”* Could you please specify the aspect or question you're particularly interested in regarding this statement? We would also like to point a potential misunderstanding: the statement of “LM agents may fail in a variety of unpredictable ways” is about the weakness of LM agents that our ToolEmu tries to expose, instead of the weakness of ToolEmu itself. > *“Also, it is unclear how biases or limitations in the evaluating LM might affect the evaluation of the LM being tested, or if ToolEmu, based on LMs, will have/incur potential risks itself.”* As described above, we do not position ToolEmu as a standalone replacement for the risk evaluation process but as a part of it to assist in quickly identifying potential failures. While we can't prove the bias properties of LM-based evaluators (e.g., the LM might have biases that prevent it from identifying certain failures), we can conduct empirical quantitative validations on them with human evaluations, as we have done in the paper.

Clarifications about terminal failure instantiations

> *“There are no clear criteria for what makes a failure “severe.”* We would like to clarify that for both our automatic evaluations and human annotations, we have provided a consistent criterion for “severe” failures as those “consequences that are significant and often irreversible such as substantial financial and property loss (exceeding $1000), irreversible damage, or compromise of personal safety" (Appendix C.2). The actual evaluation of “severity” could still be subjective even based on this criterion, and the “severe failures” in the sentence refer to those identified by the automatic evaluator. If the concerns remain, we would be open to removing the word "severe" in the sentence to avoid confusion. > *“the fact that 6 out of 7 severe failures could be instantiated on a real bash terminal is interesting but lacks statistical context. Are these failures representative?”* We have covered **all** 7 failures identified in ToolEmu without any cherry-picking on our curated test cases (we have made this point more clear in Sec 4.2). Although the sample size is small (limited by the total number of test cases, and the failure rate of the agent), we can obtain a nontrivial confidence interval of 60%-100% and the results are roughly consistent with the ~70% failure precision from human validations (Table 2). For a system that serves to surface failure modes (instead of as a single leaderboard evaluation), this result would be acceptable even in the lower range of the confidence interval. > *“It is not clear how the authors measure the time it took to instantiate these failures. Was it a straightforward process, or were there complexities that could affect the time? Are the time metrics (8 hours vs. under 15 mins) average times or one-time measurements?”* The 8-hour is a one-time assessment, where one of the authors tried to instantiate the cases from scratch. This includes establishing the environment, setting up the state, and running and examining the agent results (see Appx G.3 for details). While the time required to replicate these cases may differ based on an individual's familiarity with the environment and their relevant expertise, we anticipate that these variations will not significantly impact the relative comparisons made. The author conducting this experiment possesses familiarity with the environment and the necessary skills, such as knowledge about bash systems and programming.

The emulation realism of ToolEmu has been validated over a wide spectrum of tools and is likely to improve with future-generation LMs

> *“It's not clear that the range of plugin behaviors that can be emulated with ToolEmu matches the range of real-world software plugins being developed.”* Demonstrating perfect coverage with ToolEmu for the entire spectrum of real-world tools is challenging due to the vast and varied space of these tools. However, we would like to point out that: - We have validated the realism of ToolEmu across a diverse set of 36 toolkits, most of which are not present in previous benchmarks or have existing public APIs, demonstrating the versatility of ToolEmu. These toolkits were curated to cover representative (and high-stakes) toolkits spanning 18 different categories (Table C.5). - The range of the toolkits that ToolEmu can reliably emulate will likely expand with more capable future-generation LMs. Also, note that perfect coverage of real-world tools is *not a necessary requirement* for ToolEmu to be useful. In particular, ToolEmu would still allow us to quickly and cheaply test across different tools, including those that not have yet been developed. > *“Can ToolEmu scale to handle scenarios involving multiple plugins? Does this introduce new risk scenarios?”* We would like to clarify that ToolEmu is not limited to a single toolkit (plugin). In fact, more than 30% of our curated test cases involve multiple toolkits (e.g., Figure C.4 (a)). In the paper, we mostly selected single-toolkit examples for clarity. To make this point more clear, we have included more multiple-toolkit examples such as Figure G.5 in Appx G.1. We have conducted a stratified analysis of emulation realism of the standard emulator by the number of toolkits, shown below: | # toolkits | 1 | 2 | 3 | |---------|----|----|----| | Ratio | 94.0 (63/67) | 86.2 (25/29) | 100.0 (5/5) | This reveals that while the realism of the emulators might decrease in test cases involving two toolkits, the realistic emulation ratio remains above 85%, which we consider to be a reasonable level. The multiple-toolkit test case could indeed introduce new risk scenarios. As an example in Figure G.5, the risk of publicly sharing sensitive information could be manifested because of the combined effect of the two toolkit functionalities (i.e., Dropbox for managing sensitive files and Twitter for sharing the link).

ToolEmu serves as an exploratory tool for assisting humans in quickly testing and identifying failures at scale

> *“Not all identified failures are true failures, either because of invalid emulator behavior or invalid classification.”* It is true that not all identified failures are true ones, which we have carefully validated in the paper. However, we think it does not necessarily restrict the usefulness of ToolEmu. The main use case of ToolEmu is to function as an exploratory tool for assisting humans in quickly testing and identifying failures at scale, and a precision of ~70% would be acceptable for this intended purpose. Furthermore, the precision of identified failures is likely to improve with more capable future-generation LMs. > *“Given a range of real-world inspired plugins and user scenarios, could ToolEmu be used to identify the relative risk of real-world scenarios?”* We think this is exactly what ToolEmu is designed for. In particular, ToolEmu enables testing against specific plugins and user scenarios by design. To do so, one can just provide the tool specifications and test cases (to specify the plugin and scenarios, respectively), and run them in emulation by ToolEmu. If there are any aspects of your questions that remain unaddressed, we are eager to address any specific doubts you may have and provide further clarification if needed. > *“ How might ToolEmu's results provide insights for debugging and fixing problems?”* Due to the flexibility of ToolEmu in seamless testing, it serves as an effective tool for debugging LM agent behaviors. In particular, one can flexibly test LM agents in different scenarios and inspect identified failures in ToolEmu to understand the failure modes of LM agents. For example, from the identified failures in Figure 2, we have observed typical failure modes of current LM agents like fabrication, instruction misinterpretation, erroneous executions, etc. These observations shed light on the weakness of current agents, thereby guiding targeted improvements.

We have observed consistent failures when adjusting the prompts

> *“How sensitive is ToolEmu's findings to very minor variations in prompting and/or minor variations in emulated plugin responses? (i.e., individual word choices, punctuation, etc)”* In our development process, we have found the same conclusion when varying the prompts of our emulator. In particular, although the exact outputs of our emulator may change, the failure modes we previously found remain reproducible after minor prompt adjustments. Quantitatively assessing this sensitivity is difficult and costly due to the lengthy prompt and the need for a clear definition of “minor variations”. However, we think this level of analysis is not essential for demonstrating ToolEmu's utility.

Human evaluation is non-trivial and subjective, and we have conducted careful quality control for our human annotations

For the concerns about the validation of our automatic evaluators with human annotations, we would like to first note that our human annotation task is non-trivial and requires careful detection of agent mistakes and risky actions. Therefore, we have conducted careful quality control for our human annotations, as described in Sec 4.1 and Appx. F, which includes: - We selected 4 most qualified annotators from 25 applicants. The selection procedure involves a 30-minute interview to test their familiarity with basic programming and detailed-oriented ethics. - We ensured all the annotators were senior undergraduate students majoring in Computer Science and have completed relevant programming and machine learning courses. They have also passed a 10-example annotation test that ensures they followed the annotation guidelines before conducting the actual annotations. - All annotators undertook two rounds of annotations with the extra round for double-checking and spent about 25 hours on the annotation task. - All annotators achieved a Cohen’s $\kappa$ greater than 0.5 against authors’ annotations. Despite the careful screening, our human annotation results are not exempt from the **inherent subjectivity** (e.g., about the severity of the risks) or possible noise (e.g., humans might still make mistakes) of human evaluation (see the following discussions). > *“In Table 3, it may be overly simplistic to validate effectiveness by comparing the Cohen’s κ between human annotators and the Cohen’s κ between human annotators and automatic evaluators”* The primary factor to consider is the inevitability of subjectivity in human assessments. Take, for instance, the evaluation of a failure's severity. What one individual might deem as a mild risk, such as sending impolite emails, another could perceive as a severe risk. Given this subjective nature of evaluation, we need to use the human-human agreement as a reference to interpret the human-evaluator agreement result. Our results imply that the automatic evaluators could achieve an "average" human performance in the annotation task. We additionally conducted a leave-one-out analysis, where we computed the accuracy of our automatic evaluators and each human annotator against the majority vote of the other 3 human annotators (as the ground truth). The results are as follows: | | Safety | Helpfulness | |-|--------|-------------| | Auto (average) | 77.1 +/- 1.0 | 80.2 +/- 1.4| | Human (average) | 79.8 +/- 1.5 | 81.4 +/- 1.4| These results confirm our previous conclusion, which is included in Table D.2 now. > *“if the value of Cohen’s κ between human annotators is only less than 0.5, it raises questions about whether the annotated results of human annotators can be considered as the ground truth”* It is true that our human agreement rate is only moderate [1]. However, as described above, our human annotation task is non-trivial and involves inherent subjectivity, which may explain the relatively low agreement rate. Notably, even the Cohen’s $\kappa$ for annotations made by the authors is around 0.6. Similar results have also been observed in previous work, e.g., [2] reported a Fleiss’s $\kappa$ of 0.49 for evaluating a successful attack by their human annotators, which is comparable to 0.47 (safety) and 0.52 (helpfulness) in our case. We have included additional discussion in the Appx D.1. [1] https://en.wikipedia.org/wiki/Cohen%27s_kappa#Interpreting_magnitude [2] Ganguli, Deep, et al. "Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned." arXiv preprint arXiv:2209.07858 (2022).

We would like to ask for some additional suggestions and clarifications

> *“The paper could benefit from reorganization to enhance clarity… the frequent transitions between the main text and the appendices could be confusing. I would suggest the authors consider optimizing this structure.”* We appreciate your feedback regarding the organization of our paper. To address this, we have consolidated references to the appendices where applicable with frequent references, positioning them at the end of the paragraphs to minimize transitions. We are open to further suggestions on how we might optimize the structure for improved clarity. > *“The contribution of this work could be further improved by providing more interpretability.”* Thank you for the suggestion. However, we find the concept of “interpretability” 'to be somewhat broad. Could you please provide more detailed guidance or specific aspects that you believe need to be improved?