LalaEval: A Holistic Human Evaluation Framework for Domain-Specific Large Language Models

Summary

1. This paper proposes LalaEval, a framework designed to standardized and systematic human evaluation for domain-specific LLMs. The framework consists of 5 components including domain specification, criteria setting, benchmark dataset creation, construction of evaluation rubrics, and analysis of evaluation outcomes. 2. The framework’s efficacy is demonstrated through a case study in the logistics industry, showing how it helps elucidate performance differences among LLMs.

Rating

4

Confidence

4

Ethics flag

1

Reasons to accept

1. Constructing a standardized framework for human evaluation processes is an intriguing idea. Relevant research can help formalize the assessment procedures in the field of large language models, leading to more reliable evaluation outcomes. 2. This study provides a detailed description of each module of LalaEval, comprehensively considering the processes involved in the human evaluation procedure.

Reasons to reject

1. My major concern is that the experiments in this article struggle to demonstrate the advantages and effectiveness of the proposed evaluation framework. On one hand, the author conducted only a case study involving three models, lacking validation across a broader range of domains and models. Additionally, the experimental results presented do not clarify how LalaEval surpasses previous human evaluation paradigms. On the other hand, the author only displayed the final evaluation results and conclusions, lacking presentation and validation of intermediate steps. There is also a lack of detailed description of the human evaluation details, such as the number of evaluators involved, inter-annotator agreement, and so forth. 2. As an academic paper, I think that the framework proposed by this study does not present any fundamental innovation compared to the commonly used human evaluation processes previously established.

Summary

This paper introduces LalaEval, a holistic framework for conducting standardized human evaluations of domain-specific large language models (LLMs). The framework consists of five main components: domain specification, criteria establishment, benchmark dataset creation, construction of evaluation rubrics, and analysis and interpretation of evaluation outcomes. The paper demonstrates the framework's application within the logistics industry, presenting domain-specific evaluation benchmarks, datasets, and a comparative analysis of LLMs. The results highlight the framework's effectiveness in elucidating performance differences and guiding model selection and development for domain-specific LLMs.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

- The paper addresses a crucial research gap by providing a systematic methodology for conducting standardized human evaluations within specific domains, which is essential for the practical application of LLMs in real-world business settings. - The five-component framework is comprehensive and well-structured, covering all essential aspects of domain-specific LLM evaluation, from domain specification to the analysis and interpretation of evaluation results. - The case study of the logistics industry demonstrates the framework's practical utility and its ability to generate valuable insights for businesses looking to integrate LLMs into their operations.

Reasons to reject

- The case study focuses on the logistics industry, which may limit the generalizability of the findings to other domains. Additional case studies across various industries would strengthen the paper's claims about the framework's broad applicability. - The paper does not provide a detailed comparison of the LalaEval framework to existing evaluation methodologies, making it difficult to assess its relative advantages or disadvantages.

Summary

This paper introduce a holistic framework for the human evaluation of domain specific large language models. This framework have five main components: domain specification, criteria establishment, benchmark dataset creation, construction of evaluation rubrics, analysis and interpretation of evaluation results.

Rating

5

Confidence

3

Ethics flag

1

Reasons to accept

1. The evaluation framework for domain specific human evaluation of LLMs is beneficial to the community.

Reasons to reject

1. Certain method details are missed. For example, it is hard to tell how to construct a hierarchical domain specification for the logistics industry in Figure 1. The authors mentioned about the methods of the principle of mutual exclusivity, qualitative prioritization and so on, yet these methods are not well described and would be hard to understand if the readers do not have related background. 2. It would be better if the authors could release the dataset and evaluation code they’ve created in this work. 3. There could be more analysis to the evaluation results. For example, GPT4 underperform all the compared language models in all the results. How could we explain this phenomenon and is there any reasons that lead this result?

Questions to authors

1. How could we evaluate if the proposed LalaEval is working better than other evaluation metrics or frameworks? 2. The font in the figures is a little small and it needs to be set larger.

Summary

The paper introduces LALAEval, a comprehensive human evaluation framework designed to assess LLMs. The framework incorporates a diverse set of evaluation tasks and metrics to provide a holistic understanding of LLM's performance, addressing the limitations of existing evaluation methods.

Rating

5

Confidence

4

Ethics flag

1

Reasons to accept

1. The framework encompasses a diverse range of evaluation tasks and metrics, allowing for a nuanced assessment of LLMs' performance. 2. The empirical results showcase the efficacy of LALAEval in capturing various aspects of LALMs' behavior and facilitating fair model comparisons.

Reasons to reject

The discussion on potential biases or limitations of the proposed framework, as well as strategies to mitigate them, could be more comprehensive.

Questions to authors

1. I was questioned about the adapability of this framework to various domain applications. 2. what is the novelty of this proposed framework?

Summary

This paper proposes LalaEval, an human-centered evaluation framework for domain-specific large language models (LLMs). The core principles behind the proposed work consist of five components: domain specification, criteria establishment, benchmark creation, evaluation rubric construction and results interpretation. This work combines the proposed principles and presents a general deployable framework. The authors also present the evaluation results of 3 proprietary models fine tuned on top of ChatGLM along with Ernie Bot, GPT-4. Overall this paper is more of an industry-track position piece than a pure methodological one.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

This work is well motivated and I agree with the authors that LLMs, especially in domain-sensitive scenarios, need to have better testing protocols. This paper presents a conceptual framework that allows systematic integration of human judgment in evaluating LLM’s in-domain capabilities. From my perspective, the overall framework seems to be well structured and can be standardized in real-world applications.

Reasons to reject

From my understanding, LalaEval seems to demand significant resources, including time, trained personnel, and high-quality domain-specific data. Despite the nature of this piece being a positional one, I feel that this concept will be more complete if the authors can provide more details on the evaluations. Specifically, slightly more information on the process of selecting human evaluators, the training they receive, and how their evaluations are standardized would be beneficial. Additionally, as the authors have outlined in the evaluation framework constructions, there are multiple scoring stages. It would be beneficial to include them in the main evaluation for a more detailed per-model evaluation. Minor suggestions: - Try increasing the font size of figure 1 or otherwise the figure can be difficult to read. - Be careful about the use of \citep and \citet (e.g. first sentence of Related Work) - When describing framework specifications, avoid using specific examples as the bullet points (e.g., change "Specific Knowledge about Huolala" to "Company-specific Knowledge" in 3.3.2). - If possible, consider open-sourcing a prototype.

Questions to authors

For the main evaluation, should I believe that the lower scores of GPT-4 and Ernie Bot are due to the fact they do not have access to potentially proprietary information that is used to fine-tune the in-house ChatGLM? Additionally, how does LalaEval handle the potential subjectivity and variability in human evaluations, and what measures are in place to ensure the consistency and reliability of these evaluations?

Response to the authors

I feel much sorry that I attached the wrong review to this submission, I have changed the reviews accordingly. Sorry again for my bad mistake.