WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting

Summary

The authors propose a benchmark dataset called WorkBench that consists of a variety of tasks that are typically performed in workplace settings. Each task is outcome-centric in that the outcome is unique and unambigious to make evaluation easy and consistent. This benchmark is meant to serve as a testbed for agents to see how well they could automate business activities. They have tested 5 existing ReAct agents where Llama-2 performed poorly and GPT-4 performed significantly better and they have illustrated situations where agents can take the wrong action which could lead to disastrous outcomes (for example sending an email to the wrong person).

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

A benchmark focused on business tasks is useful for a wide variety of industries and academics The paper is well written, easy to read, and the figures illustrate the problem well The paper provides a nice analysis on how different agents perform based on the model across different tasks

Reasons to reject

The novelty of the paper is very weak as there are plenty of benchmarks out there that evaluate agents in similar sandbox environments including WorkArena (which tests agents in business settings as well) [1], and other outcome-centric benchmarks such as WebShop [6], MiniWob [4], WebArena [3], Mind2Web[4] , and WebVoyager [2]. This work does not include nor cite any of these benchmarks which are highly relevant. What does this benchmark give that others lack? If we have an agent that performs well on these aforementioned benchmarks, do we really need the proposed benchmark? [1] WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks? [2] WebVoyager: Buildingan end-to-end web agent with large multimodal models [3] Webarena: A realistic web environment for building autonomous agents. A [4] MiniWob: Reinforcement learning on web interfaces using workflowguided exploration [5] Mind2Web: Mind2Web: Towards a generalistagent for the web. [6] WebShop:Towards scalable real-world web interaction with grounded language agents.

Questions to authors

Please see the weaknesses above

Summary

This paper proposes WorkBench, a benchmark for evaluating the ability of LLMs to condition on natural language and generate programmatic API calls ("use tools") to query and modify workplace-related databases such as calendars, emails, and project management entries. A key aspect of the benchmark is that it evaluates by matching the state of the databases, accounting for the fact that there may be multiple sequences of API calls that can correctly carry out a given task. The paper benchmarks 5 LLMs, both open- and closed-source, using the ReAct prompting strategy, and performs an error categorization.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

S1) The benchmark is very well motivated: a benchmark for workplace-oriented tasks is a good step toward developing LLM agents for real-world economically impactful tasks, whereas previous works have mainly focused on consumer tasks or very specialized domains. S2) The outcome-centric evaluation used here is an important feature of the benchmark, given that there are multiple possible ways that a task could be carried out. This addresses a key limitation of many existing benchmarks that evaluate using API call match. S3) The paper is overall well-written (although I have a few questions about the evaluation, see below). S4) The paper evaluated a range of reasonable models and did some interesting initial error analysis of model performance.

Reasons to reject

W1) It's a bit hard for me to gauge the difficulty of the dataset: - The paper doesn't perform a human evaluation to see how easy or hard the examples are for people to carry out. This would be very valuable to make sure that there is some headroom beyond the performance currently achieved by GPT-4+ReAct (and see below for a concern about this setup), although given the task descriptions I expect that there will be some headroom. - From Figure 4, ~72% of the examples require 0 or 1 actions to solve, and the performance of models on these examples is much higher (from Table 7 in the appendix) than the tasks that require more actions. - From Figure 6, I have some doubts about how many of the errors of GPT-4 are serious errors that will be productive for future research to resolve. Many of the errors (32%) are "Incorrect without side effects", which if I understand correctly are all failures of the API calls or of the ReAct prompting itself (by simply not generating an ACTION keyword, if I understand Sec 4.4 correctly). It would be helpful to try simple baselines on top of GPT-4 that do things like re-sampling from the model if ReAct or the API calls fail. How much closer would this get to 75% (43+32) accuracy with GPT-4? However, this isn't an absolutely crucial weakness as the benchmark is still challenging for open-source models. W2) There are some missing comparisons to other works on LLM agent benchmarks / API calling. The below I think are the most relevant given this paper's focus on tool use and task automation in the introduction: - AgentBench, Liu et al. 2023 (which is cited in this paper, but in a different context) - Mind2Web, Deng et al. 2023 - WebArena, Zhou et al. 2023 - VisualWebArena, Koh et al. 2024 (but was released in Jan 2024, close to the CoLM deadline in March) - SMCalFlow, Andreas et al. 2020 In particular, WebArena and VisualWebArena use test-based evaluation which, like the outcome-centric evaluation proposed in this work, recognizes that multiple action paths can produce a correct outcome. So I don't think that the outcome-centric evaluation (while important!) is a key novelty of this work, if I'm understanding it correctly. But, I don't think this is a crucial weakness, as the focus on workplace-relevant tasks is important.

Questions to authors

Q1) From Figure 3, it seems that the ground-truth outcome of a task is a subset of the items in the database (e.g., all events in the calendar after deleting the named event in the Fig 3 example). Is this correct that the ground-truth is a subset, or is it the entire database? If it's a subset, is the evaluation robust to the agent taking incorrect actions in other parts of the database (e.g. deleting a customer from the CRM database)? How does this interact with the "side effects" metric? Q2) Can you say more about the "Failed to follow ReAct" subcategory of Fig 6b? This seems to be a very large fraction of the overall errors of GPT-4 on the benchmark (~16%, from combining Fig6a and Fig6b) and if this were easily fixable then performance of GPT-4 on the benchmark would be quite a bit higher. Q3) The error analysis in Figure 6 indicates 32% "Incorrect without side effects". From the definition of side effects, does this mean that 32% had no effect on the environment (e.g., the API calls failed)?

Summary

The paper proposes a new dataset called WorkBench for evaluating the how well agents can perform business-related tasks, such as sending e-mails, given a natural language instruction from the user. The instruction could be simple (such as "cancel my next meeting with Carlos") or quite complex ("if our website page views fell by more than 10%, then schedule a meeting; otherwise, send an email"). To evaluate whether the agent performed the correct actions, the authors provide a simulator for the actions and propose to measure whether executing the actions results in the correct changes to the environment (i.e. the correct updates to the databases). The authors evaluate the dataset on several language models using the ReAct framework.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

- The paper addresses a timely and relevant topic. - Even though the paper focuses on proposing a new dataset, it still evaluates the dataset on multiple models to provide relatively comprehensive results. - The authors annotated all of the tasks with a detailed way to measure whether the task was performed correctly, ensuring a more precise evaluation.

Reasons to reject

- The outcome-centric evaluation has several limitations which seem under-studied in this work: - The agents may perform actions which coincidentally happen to result in the same outcomes as what the user intended given the provided databases, but would not produce the desired result when executed in a different environment. - QA-style tasks, not just performing actions, are still important for an agent assisting users in a workplace setting. The outcome-centric evaluation does not seem to be able to cover this use case, because it focuses on evaluating changes to the databases, and not what information is returned to the user. - The design of the dataset focuses on complex user requests specified in one turn, which is unnatural; a dialogue-style interaction where the user can use multiple turns, and also correct errors from previous turns, would be better. - The use of templates to create the dataset may have resulted in insufficient linguistic variation, which is not discussed in the paper.

Thanks for the response!

Thanks to the authors for the response, which concisely addressed all of my main concerns. In particular, it's a great point that the proposed approach tests for unintended side effects (which many other benchmarks don't). Thanks, too, for the upper bound on GPT-4 performance and the example of the logic failure example (which, while it too could potentially be fixed with resampling, I agree is a categorically different type of error than the syntax ones) -- these were helpful! The major remaining limitation in my mind (as the other reviewers also pointed out) is the lack of acknowledgment of related work. But, I expect that this could easily be addressed in the camera ready especially since you have a nice description of the differences from this work in the rebuttals here. I've raised my score to a 7.