Efficient multi-prompt evaluation of LLMs

Summary

The authors introduce a novel method called PromptEval which permits efficient multi-prompt evaluation of LLMs across different prompt templates with a limited number of evaluations.

Strengths

- Theoretical guarantees that PromptEval has desirable statistical properties such as consistency in estimating performance distribution and its quantiles. - Large-scale study of prompt sensitivity of 15 popular open-source LLMs on MMLU.

Weaknesses

N/A

Questions

N/A

Rating

8

Confidence

1

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

This paper introduces PromptEval, a novel method for efficiently evaluating the performance of large language models (LLMs) across multiple prompt templates. They propose a statistical framework based on Item Response Theory (IRT) for estimating LLM performance distribution across many prompts using limited evaluations and theoretical guarantees on the consistency of the proposed estimators.

Strengths

1. The paper addresses an important and timely problem in LLM evaluation, recognizing the limitations of single-prompt evaluations and proposing a novel solution. 2. The theoretical foundation is solid, with clear assumptions and complete proofs provided in the appendix.

Weaknesses

1. While the authors compare different variations of their method, they don't explore alternative modeling approaches beyond IRT. It would be interesting to see comparisons with other potential frameworks. 2. While the method saves evaluations, the paper doesn't discuss the computational requirements of the estimation process itself, which could be relevant for very large prompt sets.

Questions

1. How sensitive is PromptEval to the choice of initial prompt set? How might this impact its use in practice? 2. How does the computational complexity of the estimation process scale with the number of prompts and examples?

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors have provided a comprehensive discussion of limitations in the appendix, which is commendable.

Summary

This paper introduces PromptEval, an efficient multi-prompt evaluation method for LLMs, showing its statistical consistency and effectiveness across benchmarks (MMLU, BBH, LMentry) and studying prompt sensitivity in 15 open-source LLMs.

Strengths

- The authors conducted a comprehensive theoretical analysis to verify that PromptEval possesses ideal statistical properties, and carried out extensive experiments using three benchmarks (MMLU, BBH, LMentry) and fifteen open-source large language models (flan series, llama series, mistral series, etc.) - They introduces five variations, comparing the baseline under different budgets (i.e., different sampling rates) and various quantiles, demonstrating the significant effectiveness of PromptEval.

Weaknesses

- The current experiments have only been conducted on open-source models. I look forward to experiments on some mainstream closed-source models (such as the GPT series or the Claude series) and explore corresponding variant strategies. - This method PromptEval requires that a large number of active prompt temples exist in its benchmarks, but this is often not satisfied. Does this method work if there are only a few temples in benchmarks? This requires authors to carry out some experiments to prove. - The construction of the baseline is relatively simple, and authors need to compare it with some relevant methods (like TRIPLE[1]) to highlight the advantages of PromptEval. [1] Chengshuai Shi, Kun Yang, Jing Yang, and Cong Shen. Efficient Prompt Optimization Through the Lens of Best Arm Identification, arXiv preprint arXiv: http://arxiv.org/abs/2402.09723.

Questions

Please refer to my comments in "Weaknesses".

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The author pointed out the limitations in the Appendix.

Reply by Reviewer

Thank you for your detailed rebuttal. Your response has addressed my major concerns. I will also engage in further discussion with the other reviewers to ensure we consider your clarifications.