MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures

Summary

The authors propose the MixEval to match the real-world human queries with existed benchmarks. MixEval is a two-stage benchmark reconstruction pipeline consisting of (1) wild query detection, and (2) grounding existing benchmarks in the mined queries. The authors match each crawled web user query with its most similar query in the benchmark pool and the corresponding ground truth answer to align benchmark queries with real-world queries. In order to improve the benchmark’s ability to distinguish strong models, the authors derive a challenging subset from MixEval, which is called MixEval-Hard.

Strengths

1. The authors match each crawled web user query with its most similar query in the benchmark pool and the corresponding ground truth answer to align with the human perferences. 2. The experimental results demonstrate that the MixEval and MixEval-Hard are highly aligned with Chatbot Arena, outperform singular benchmarks.

Weaknesses

1. In Section 3.2, the authors use dot product to match the query with the original benchmark, but do not explain how to use new queries in the test process and whether the queries are rewritten adaptively to match the ground truth answers. Why the answers from the original benchmark can be considered as the answers for the new queries? 2. In Figure 5, the description of the experimental setup is missing. It is not clear 0-shot or 5-shot is used in Figure 5, and how the inputs of "Mixed" and "Original" are formated for the models. 3. It is better to demonstrate the "User query" and "Ground truth-benchmark" processes are effective for constructing benchmarks through the ablation studies.

Questions

1. Why can queries from different batches of the same distribution mitigate the contamination problem, instead of sampling queries from different distributions? 2. Why the performance of the Benchmark-level Mix in Figure 5 is lower than the average performance of the Mixed benchmarks?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed detailed limitations in the Appendix A.

Summary

The paper reconstructs a new benchmark named MixEval by matching queries collected from the internet with existing benchmarks. This new benchmark aligns with the distribution of human preferences, reflecting the real distribution of queries on the internet. Additionally, considering the overlap and difficulty among different benchmarks, the paper introduces MixEval-Hard by recalculating model scores across various benchmarks. Both MixEval and MixEval-Hard can be updated quickly. Finally, the paper demonstrates that MixEval aligns better with Arena Elo compared to other benchmarks, and the mixing operation enhances the alignment of other benchmarks with Arena Elo.

Strengths

1. The experiments are comprehensive, providing substantial results that validate the proposed method's ability to align well with the Arena Elo benchmark. 2. By utilizing queries from the internet to shift the distribution of existing benchmarks to reflect real human preferences, the approach helps prevent large models from overfitting on existing benchmarks. Additionally, the paper offers insights into the consistency between benchmarks, indicating that alignment results are influenced not only by query distribution but also by question difficulty and density. 3. An interesting visualization method is proposed, effectively reflecting the main aspects evaluated by the existing benchmarks.

Weaknesses

1. Given that alignment with Arena Elo is used to measure the degree of alignment with human preferences, why not use data directly from Arena Elo when constructing MixEval, or even create a subset sampled from Arena Elo (considering Arena Elo directly reflects human preferences)? 2. To avoid models overfitting on existing benchmarks, the paper updates the benchmarks by drawing new queries from the internet. However, I believe overfitting occurs due to models overfitting on a fixed benchmark. Without changing the benchmark, the overfitting issue will not be resolved. Therefore, a method to determine if a benchmark is overfitted should be designed to add LLM benchmarks to the pool. Additionally, Table 2 does not demonstrate that this method alleviates overfitting. 3. Queries drawn from the internet might have already been seen by LLMs, so the retrieved samples from existing benchmarks might be simpler. The authors could exclude such factors during query extraction. 4. Some symbols are not explained, such as "j" in line 187, "tau" in the formula between lines 187 and 188, and "lambda" in line 189.

Questions

Please see above.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes.

Summary

The paper introduces MixEval, a new benchmarking framework designed to overcome the limitations of traditional benchmarks and LLM-as-judge methods for evaluating LLMs. By leveraging web-mined user queries and matching them with existing benchmark queries, MixEval aims to offer a fast, efficient, and dynamic evaluation method that aligns closely with real-world human preferences. This framework promises significant cost and time savings and shows a high correlation with human preference leaderboards like Chatbot Arena.

Strengths

1. This paper tackles the important evaluation problem timely. MixEval introduces a fresh approach to bridging the gap between real-world user queries and efficient evaluation. By using web-mined queries, it aims to better reflect actual user interactions. 2. The framework is efficient, could do dynamic updates and has high Correlation with Human Preferences. 3. The paper goes above and beyond by providing extensive analysis and comparison with other popular LLM benchmarks, giving valuable insights into the strengths and weaknesses of different evaluation methods. 4. The paper is nicely written, with figures and tables well-organized. In particular, Figure 2 stands out for its clarity and effectiveness in presenting complex information.

Weaknesses

1. Pipeline Brittleness: a. Web User Query Detection. The web user query detection phase has an overabundance of negative examples, which does little to help distinguish positive examples. b. Benchmark Mixture. Additionally, the sentence transformer used in the benchmark mixture phase has limited performance. Have you tried other sentence embedding models? Also it's unclear how many ground-truth LLM benchmarks can be accurately matched to mined queries. c. The error rates for each module and the accumulated error rate are not provided. 2. Single-Language Focus: The all-mpnet-base-v2 model used in the framework is designed for English only, raising concerns about its adaptability to different linguistic and cultural contexts.

Questions

1. Could you provide detailed qualitative and quantitive analysis of module Web User Query Detection and Benchmark Mixture? 2. It seems that the query of MixEval-Hard is longer than MixEval, is there any distribution difference between them?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

See weaknesses.

Summary

This paper introduces MixEval, a new approach/benchmark to evaluate LLMs effectively in real-world scenarios. Traditional benchmarks often miss the comprehensiveness and subtlety of actual user queries, while existing methods like LLM-as-judge benchmarks are difficult to scale up. MixEval addresses these issues by using user queries mined from the web and aligning them with similar queries from established benchmarks. The key advantage of MixEval is its efficiency and dynamic nature. It achieves a high correlation with Chatbot Arena (0.96 Spearman correlation) but requires only 6% of the time and cost compared to mainstream benchmarks like MMLU. MixEval can also be updated quickly, significantly reducing the risk of benchmark contamination. The paper also introduces MixEval-Hard for better differentiation among strong models. Through comprehensive analysis, the authors demonstrate that MixEval offers accurate, less biased, and more dynamic evaluations, providing a potential scalable solution for real-world LLM assessment.

Strengths

1. Aligning web queries with mainstream benchmarks to simulate real-world user preferences is a novel approach. The authors cleverly transform the challenging task of evaluating open-ended queries into a benchmark mixture with groundtruth-based results, which is an interesting idea. 2. MixEval is an effective alternative to ChatBot Arena as it can scale up, dynamically update, and has lower costs. 3. The authors provide a comprehensive analysis of various LLMs on MixEval, demonstrating high correlations with ChatBot Arena.

Weaknesses

**Major Issues**: 1. MixEval dynamically updates by mixing popular benchmarks (e.g., MMLU, BoolQ, GSM8K), which may not mitigate contamination. Most of these benchmarks are saturated and suffer from contamination. Although the authors claim they will "dynamically expand our benchmark pool with newly released benchmarks to further enhance the mixed benchmark distribution," this does not address the root issue of contamination. **Minor Issues**: 1. While the topic distributions in Figure 2 are impressive, most of these benchmarks were not designed to simulate human preferences, hence the skewed topic distributions may be intentional for evaluating specific tasks, domains, or capabilities. Therefore, the section title "LLM Benchmarks are Biased from Realistic User Queries and Preferences" seems biased, implying these benchmarks are meant to measure human preferences. 2. Another potential issue is the assumption that ChatBot Arena serves as the groundtruth for calculating correlations. As noted by the authors in lines 127-133, ChatBot Arena itself is biased. Therefore, even though MixEval has a larger user base, it might overfit to ChatBot Arena to some extent, making it more of an alternative rather than a more accurate evaluation of user preferences.

Questions

I am a bit confused about Table 2. The authors briefly mention "periodically update" in the main text, but I did not fully understand the details of this update. Could you explain this further?

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

In the appendix, the authors address some limitations in a Q&A format, such as the potential biases in MixEval. My views on limitations are reflected in the weaknesses above.

Thank You for the Response!

Thank you for your response and the additional experiments, which have led me to increase the score from 6 to 7. I agree that MixEval can mitigate contamination, but I still do not believe it fundamentally solves the problem, as it still heavily relies on existing benchmarks. I disagree with the authors' explanation for Concern 2, as many benchmarks were not originally designed to measure human preferences, and human preferences are just one aspect of the many evaluations for LLMs (although a very important one). Technical reports for all LLMs release results from numerous benchmarks to give the community a comprehensive understanding of their capabilities. Calculating correlations between all these benchmarks and human preferences introduces some potential biases, such as assuming they should align with human preferences, which is not necessarily the case. Despite these issues, which I consider minor, I personally recommend the acceptance of this paper. Good luck!

Thanks for the response!

Thank you for your response. The user's query is multimodal, so it is reasonable to use a mixed benchmark. My concern regarding the leakage issue has been partially resolved. Additionally, concerning Concern 3, although the queries in the benchmark are relatively rare on the entire web, it's possible that all queries on the web have been used as training data for the query-matching model. Therefore, I wonder if using these queries to match those in the benchmark might be more likely to match the polluted queries rather than with queries that the model has never seen. However, based on Table 1 in the final rebuttal response, it seems that the query matching does not introduce this bias. Thank you again for your response, and I will raise my score.

Thank you for your response!

Thanks for the response. Looking forward to your final version.