SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models

Summary

Authors create SheetCopilot that takes input as a natural language task and then controls spreadsheet to perform the task. To provide an interface between natural language and the spreadsheet, author use a set of atomic actions that abstract spreadsheet functions. SheepCopilot uses state machine to create spreadsheet actions, modify them, and apply them to the spreadsheet. Authors also present a dataset to evaluate their tool. SheetCopilot outperforms baseline approaches

Strengths

- Authors propose an approach to interface LLMs with spreadsheets potentially automating a large number of repetitive tasks - Intermediate language of atomic actions provides a good output structure for LLM - State machine based processing of tasks improves the results - Atomic action name substitution with synonyms that are far away from the official names in an embedding space is interesting approach (but see weaknesses) - Test dataset is useful for evaluating SheetCopilot and similar systems - Ablation studies show clearly the contributions of different parts of the system - Positive results compared to VBA generation baseline - Paper is well written

Weaknesses

- Paper contributions are moderate: - Intermediate language similar to atomic actions is a known approach in software engineering, similar to macro or scripting languages. - Atomic actions have to be implemented in spreadsheet software which requires more effort to adopt this approach compared to existing interfaces to spreadsheets. (This is not a major weakness though, especially if current interfaces to spreadsheets are not well suited or universal). - Atomic action name substitution with synonyms seems like a hacky approach, a bit similar to security-through-obscurity. It may have negative consequences (e.g. making it harder for LLM to connect tasks and actions) and it is not clear if this is should be used or relied upon in general. I have read the author’s rebuttal. The rebuttal partially addressed my concerns about atomic actions.

Questions

Why do certain tasks fail? Are the failures in certain categories due to the tasks, some limitations of LLMs, selection of atomic actions, spreadsheet functionality, too simple state machine? (Also see limitations comments). I have read the author’s rebuttal. The rebuttal addressed my concerns about understanding of why certain tasks fail by providing failure analysis.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

It would be good to have a more in-depth understanding why certain tasks fail and where are the boundaries of LLM spreadsheet interactions. It is good that authors classified tasks into six categories and observed different failure rates across them. However, it would be good to go further and understand the failures in more depth (also see questions). I have read the author’s rebuttal. The rebuttal addressed my concerns about understanding of why certain tasks fail by providing failure analysis.

Summary

This paper introduces an LLM based spreadsheet manipulation task system from high level human language to spreadsheet manipulation tasks. The work is in the area of tool augmented LLM systems where LLM systems are used to create a chain of actions representing a complex manipulation task in a spreasheet system. The authors create a curated dataset consisting of a list of realworld spreadsheet tasks. They also build an assessment system. They study 3 popular LLMs - GPT-4, GPT-3.5turbo and Claude. They surprisingly find that GPT3.5turbo to be the best performing and also through ablation studies find that close-control action loops improve functional correctness, bringing in external docs improve correctness and surprisingly find that using synonyms far from the actual names also improve

Strengths

1. The authors take on the challenging problem of automating complex modification task in spreadsheets rather than formula repair or generation. If it works can be of big benefit to spreadsheet users. 2. The authors create a curated dataset of tasks in a systematic way. 3. They also create a systematic evaluation system for evaluating the performance of the different popular LLMs in the context of the spreadsheet software system.

Weaknesses

1. I would have liked to see a better discussion on the task colletions, the Q&A pair collection, how complex these tasks are, their categorization, why they are multicategorized, what do those "practical realworld spreadsheets" look like etc. 2. The techniques using LLMs are fairly straightforward with minor modifications like using synonyms (not sure why this works) adding documents etc. What is specific to the worksheet manipulation scenario that makes LLM easy or hard or different to work with. Lack appreciation of that. 3. Further investigation into why GPT-3.5Turbo is better than even GPT-4 would be useful as it is counterintuitive. 4. Instead of just exec@1, pass@1 softer metrics would be useful in understanding where these LLMs make mistakes and where they are good. 5. Instead of using a tag cloud for dataset evaluation something detailed metrics would have been more insightful.

Questions

1. Reference to Table 1 seems off. The citation says something and the table is about something else. 2. Is the dataset for creating tasks and benchmarking available to the public to compare. 3. Did you try smaller open source LLMs and see if they can be useful or how far you can get with them. 4

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

1. There is not much study on the nature of the dataset curated. But the paper is just at the beginning of something interesting. While it is unlikely that one may see issues with spreadsheet data in discrimination or bias or fairness it would be good to address them.

Summary

The paper proposes a benchmark and a framework based on observe-propose-revise-act for tackling spreadsheet problems.

Strengths

- The paper is very well written and the ideas are neatly presented with figures and tables. It was a pleasant read! - The ablation studies are quite interesting. It is interesting to see the effects of external documents, usage examples, state of the spreadsheet, and error messages separately. I also liked the ablation with the synonym substitution for official atomic action names. - The paper proposes a comprehensive framework of observe-propose-revise-act, which even though shown to work on a restricted domain of spreadsheets, can be used as inspiration for other systems, though it is not clear how scaling it up to more complex scenarios will work in practice.

Weaknesses

- My main concern is over the benchmark creation process. Since the problems are picked up from a public domain site, I think there is a high possibility that these LLMs might have already seen this data during pretraining. Under this assumption, it becomes hard to understand the generalization capability of the proposed framework for unseen tasks. The authors don't discuss any deduplication efforts they performed to ensure that the performance of the shown metrics is not bloated. - Parts of the framework have been proposed by other works as discussed in the related works. However, I would consider the framework as a whole as a novel contribution. At the same time, I believe that the tasks tacked by the paper can be considered too simplistic for these LLMs ( especially since it is shown that the LLM achieves 100% performance on some categories of these tasks). It is not clear to me how the proposed framework would scale to more complex settings like a Copilot for programming where it is difficult to retrieve the relevant state of the environment, external documentation, or usage examples. Even if there exists a way to get this information, it is unlikely that it will fit within the context length available in the input prompt. - From my understanding, there is no timeout used and the LLM is queried multiple times until it gets the right answer. If this were the case, this is not a realistic assumption when deployed in practice. Users would not wait for long periods until they get the required task accomplished.

Questions

- It would be good to discuss the performance-latency tradeoff in terms of #queries to the LLM, and the memory and compute used for inference. - It would be good to have a user satisfaction metric as well, e.g. if the users were satisfied by the time it look for the model to accomplish the task, etc. - Line 112: What are LLM-based filters? - It would be good to show the improvements brought by the framework on top of relatively smaller LLMs.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

There is no limitation discussed in the main paper. A discussion about the increased computing requirements during inference and generalization to complex settings should be included.

Summary

This paper introduces SheetCopilot, a model that aims to generate step-by-step executable command sequences for software control according to the natural language description. Besides, a benchmark dataset for evaluating software control tasks is collected. Experimental results based on the dataset are reported.

Strengths

- Generating step-by-step executable command sequences for software control according to the natural language description is a valuable problem. - The proposed framework SheetCopilot takes a set of atomic actions as an abstraction of spreadsheet software functionalities and contains a state-machine-based task planning framework for LLMs to interact with spreadsheets, aiming to translate high-level task description into executable command sequences, which generally makes sense and is actually novel. - Experimental comparison against baselines (including the state-of-the-art LLMs like GPT-3.5 & GPT-4) has shown the performance advantage of the proposed method. Besides, the ablation study indicates that the design of the proposed framework is beneficial for improving performance. - In addition, a comprehensive dataset containing 221 spreadsheet-related tasks is collected in this work for performance evaluation, and this dataset could be further served as a benchmark dataset in this area.

Weaknesses

- Reporting the stability test results through the line chart may be clearer. Besides, there is still some space to report more empirical evaluation results.

Questions

Is the collected dataset publicly available?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

None.

Thanks for the comments

I appreciate the comments and additional discussion provided by the authors.