CHOPS: CHat with custOmer Profile Systems for Customer Service with LLMs

Summary

The authors propose a framework for Retrieval-Augmented Generation based on LLMs in the domain of customer support. The framework utilises a series of smaller and larger LLMs to retrieve information from a database and to serve this information to the customer. The database is in the form of pdf guides and a MySQL database with data on users, schools, exams and answers. The developed framework is evaluated on the customer service of a Physics olympiad for teachers and students. A series of models and frameworks is evaluated, and the dataset used for evaluation is published.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

1. A realistic and useful scenario is evaluated. In the examined scenario, a customer service agent is facilitating information retrieval between a customer and a database. 2. The comparison of the different frameworks is highly informative, especially the compartmentalization of the different modules, such as the use of the Classifier, which greatly increases performance.

Reasons to reject

1. Section 5.1 (Metrics) should be expanded. The evaluation methodology needs to be clearly set before the experiments and not in the Appendix. Both the metrics and the motivation behind them needs to be expanded upon in the main paper. 2. Open-source baselines are missing. As mentioned by the authors, the Classifier and Verifier deal with easier tasks. It would be interesting to examine open-source LLMs for these tasks. (amended in rebuttal) 3. Security and inference time are not discussed. The proposed framework does not currently deal with malicious user behaviour, which needs to be a greater focus. Further, for the proposed framework to be adopted, a study and comparison on inference cost is pivotal. (amended in rebuttal) 4. The paper needs some further editing, there are multiple visual miscues present, such as: - Some citations are missing and some have spacing issues (no space between proceeding text) - Grammatical typos, such as: "but high accuracy are needed" - Figures 1, 2 and 4 are too small to read

Summary

This paper discussed the growing use of Large Language Models (LLMs) such as GPTs and LLaMas in businesses and software platforms for customer service. The authors identify a gap in the integration of LLMs with customer profiles and operational capabilities in customer service models. To address this, they propose an LLM agent called CHOPS that efficiently uses existing databases or systems to access user information and provide accurate responses while avoiding harmful operations. The authors introduce a practical dataset, the CPHOS-dataset, collected from an online platform that organizes simulated Physics Olympiads for high school teachers and students. They propose a three-agent architecture for customer service tasks: Classifier, Executor, and Verifier (C-E-V). The Classifier classifies user requests, the Executor provides responses or executes operations, and the Verifier assesses the validity of commands executed by the Executor. The paper also includes experiments to validate the performance of the proposed CHOPS architecture using the CPHOS-dataset. The results demonstrate that the CHOPS architecture can enhance or serve as an alternative to human customer service. The authors conclude by emphasizing the need for more datasets with QA pairs and Database for Customer Service to further evaluate the effectiveness of the CHOPS-architecture.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

The paper addresses a significant gap in the integration of LLMs with customer profiles and operational capabilities in customer service models. This is a relevant and timely topic given the increasing use of AI in customer service. The introduction of the CHOPS architecture and the C-E-V model is innovative and could potentially change the way customer service is handled in businesses. The authors have done a commendable job of validating their model using the CPHOS-dataset. The results demonstrate the effectiveness of the proposed architecture.

Reasons to reject

1. The paper could benefit from a more detailed explanation of the CHOPS architecture and the C-E-V model. The current description may not be sufficient for readers unfamiliar with these concepts. 2. The authors have used the CPHOS-dataset for validation. However, the effectiveness of the model in real-world scenarios remains untested. 3. The authors could have included a comparison with other existing models. This would help readers understand how the proposed model stands against the current state-of-the-art.

Summary

The paper proposes a framework called CHOPS (CHat with custOmer Profile in existing System) that aims to integrate large language models (LLMs) into customer service systems that involve guiding files and databases. The key components are: 1) A new dataset called CHOPS-dataset derived from a real-world online platform for physics Olympiads, including a database, PDF guides, and user questions/instructions. 2) A classifier-executor-verifier architecture where the classifier determines what information (database, guides) is needed, the executor generates a proposed answer/operation using that information, and the verifier checks if the executor's output is valid. 3) Using different LLMs for each component to balance accuracy and cost - e.g. a powerful LLM like GPT-4 for the executor but cheaper models like GPT-3.5 for the classifier/verifier. The authors conduct extensive experiments showing their architecture can achieve high accuracy (98%) at reasonable cost on the CHOPS-dataset by combining LLMs in this way.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

1. The paper proposes a novel application of integrating LLMs into real-world customer service systems with databases and guidance in a accurate and cost-effective manner. 2. New dataset (CHOPS-dataset) derived from practical scenarios to facilitate research in this area. 3. Thorough experiments evaluating the different components and comparing performance/cost trade-offs. 4. The classifier-executor-verifier architecture is a promising approach that could generalize beyond just customer service.

Reasons to reject

1. Results are solely on the CHOPS dataset, which is quite specific and limited. More experiments on other customer service domains would strengthen the claims. 2. Limited analysis of why the classifier-executor-verifier works well and what kinds of errors/failures occur. 3. The database/API setup is complex and not easily reproducible from the paper.

Questions to authors

1. The paper mentions iterating the pipeline up to 5 times if the verifier rejects outputs. How often do multiple iterations end up being required in practice? 2. The results focus on English inputs/outputs. How might you handle customer service scenarios requiring other languages? Would the architecture need to be modified?

Thanks for reviewer responses; Open for further discussions

Dear Reviewers, Thank you for taking time to do reviews and responses! We hope our previous feedback provide more supplementary information about our work and address your concerns. We are also looking forward to possible further discussions in the remaining discussion period!

Summary of Revisions

We sincerely thank all reviewers for their detailed comments, which are very constructive for our work. In our work, we proposed a real-world Customer Service dataset with database, guiding files and QA pairs. We also proposed the CEV agent-based architecture for utilizing LLMs on Customer service and evaluated it on our proposed dataset. We hope our work may bring insight for LLM applications in Customer Service in the aspect of agent architecture and encourage the proposal of more real-world datasets with databases, APIs, related guiding files and other related materials as well. The reviewers generally hold positive opinions to our paper, in that our work proposes a novel and timely application of LLMs in the real-world scenario of Customer Service, our experiments are informative that show the effectiveness of our proposed architecture on performance/cost trade-offs, and our proposed architecture provide insight that may generalize beyond Customer Service. The reviewers have also raised detailed and constructive concerns that help us improve our work. Compared to the original version, our new version of paper will feature enhancements in the following aspects: **Supplementary experiment results and analysis** 1. The table for Comparison with baselines will be added to the main paper. (Reviewer d31m, Reviewer T4SG) 2. The table for Performance of open-source llms as Classifiers as well as analysis about shotcomings of Llama2 will be added. (Reviewer T4SG) 3. More analysis about failure case and iteration number required in practice will be added. (Reviewer Utie) **More detailed explanations in the main paper** 4. More detailed explanation about the CEV model will be added in the main paper. (Reviewer d31m) 5. More details about experiment settings will be added in the main paper. (Reviewer T4SG) 6. More details about estimation methods for inference cost will be added in the main paper. (Reviewer T4SG) **Amendment for Miscues presents** 7. Miscues presents, typos will be corrected and figures will be redrawn for readability. (Reviewer T4SG) We will also modify current codebase to simplify setup procedure for better reproducibility. (Reviewer Utie) The valuable suggestions from reviewers are very helpful for us. We would be glad to have further discussions.