WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild

Author Response (1/2)

Thank you for liking our paper and giving thoughtful reviews, as well as recognizing our strengths. We appreciate your detailed review of our submission and your valuable suggestions, which have helped us identify areas for improvement. Below, we address your concerns in detail. --- ### 1. Novelty of the Work **Respectfully, we disagree with the assessment that our contributions are limited in novelty.** While we do use large language models as evaluators, similar to other benchmarks such as AlpacaEval, we have introduced several significant innovations that set our work apart. **First, we leverage a large corpus of real-world user data to identify challenging tasks for authentic evaluations.** **Unlike previous benchmarks, our focus on difficult, real-user queries provides a realistic assessment of LLM capabilities in handling complex and diverse situations.** For example, many questions in AlpacaEval are quite simple and do not effectively differentiate model capabilities. Questions such as "What is the capital of Australia?" or repetitive recipe-related inquiries (e.g., over 20 similar recipe questions) do not challenge the model's abilities adequately. In contrast, our approach in WildBench aims to evaluate models with more complex tasks, thus providing a better understanding of model robustness. Second, regarding automated evaluation, we adopt a better approach compared to others like AlpacaEval. **Instead of simply choosing between response A or B without providing reasons, we use a Chain of Thought (CoT) methodology, where the evaluator analyzes each response step by step, noting strengths and weaknesses before arriving at a final decision.** This detailed analysis adds transparency and depth to the evaluation process, which is not commonly employed on such a large scale in previous benchmarks. **Moreover, we use a checklist to guide the evaluation process, which ensures the focus is on the model's problem-solving ability rather than overfitting to GPT preferences.** This also helps evaluators pay closer attention to different facets of the responses, making the evaluation more holistic. **Finally, we incorporate multiple baseline models to make pairwise evaluations more accurate and reliable, ensuring robustness across different model comparisons.** The checklist plays a crucial role in our evaluation process by ensuring that evaluators systematically consider multiple aspects of a response, such as relevance, coherence, and factual accuracy. This structured approach helps avoid overfitting to GPT preferences and focuses on a comprehensive assessment of problem-solving abilities. By using a checklist, we provide a more reliable and thorough evaluation, which ultimately enhances the robustness and fairness of our benchmark. --- ### 2. Handling of Length Penalty **We appreciate your feedback on the handling of length penalties.** We would like to clarify that our current approach only applies length penalties under specific conditions: when the response length exceeds a certain threshold, and when the evaluation outcome is "slightly better" rather than "much better" or "slightly worse." **We agree that this approach may not always be entirely accurate, as longer responses can sometimes be preferable.** Therefore, we allow users to adjust the length penalty parameter (K) to create a personalized evaluation. If a user prefers longer or shorter responses, they can modify K accordingly, providing flexibility for different evaluation preferences. **In future versions, we also plan to implement more comprehensive solutions for handling length penalties.** For example, during annotation, we intend to have our evaluators label multiple aspects of a response, such as style, factuality, and helpfulness. This approach will help us better differentiate stylistic elements, especially advantages related to response length, from other important evaluation dimensions. (to be continued)

Author Response (2/2)

### 3. Compared to Existing Benchmarks **Our benchmark focuses on the "wildness" of user interactions, which we believe is a significant differentiator.** Our evaluation results, based on real user queries, demonstrate a closer alignment with human-based evaluations, such as those seen in ChatBot Arena. Unlike ChatBot Arena, which relies on human evaluators and lacks the scalability needed for rapid assessment of developing models, **WildBench provides an automated yet reliable evaluation framework that mirrors human preferences.** This also allows us to generate synthetic feedback efficiently, which is not possible with purely human-based evaluation methods. We also acknowledge your point regarding the results involving SimPO. In many cases, models or methods might overfit to certain superficial features, which can deceive GPT-based evaluators, especially in simpler methods like AlpacaEval that do not use our checklist and CoT-based judgment. For example, SimPO or similar methods and models might achieve very high scores on AlpacaEval (sometimes an 8B model can even outperform GPT-4), yet they fail to replicate similar results on WildBench. **This difference highlights how our data and evaluation methods better align with real user experiences, avoiding overfitting to GPT-4 preferences for simple questions, and instead focusing on improving the true helpfulness of the model.** This is the effect we aim to achieve with WildBench. --- ### 4. Beyond a Dataset and Evaluation Work **We believe that WildBench is more than just an evaluation framework; it is also a tool for modeling synthetic feedback for natural and challenging tasks from real users.** **One of our major contributions is demonstrating that WildBench evaluations are strongly correlated with human judgments, meaning our automated method can effectively simulate human preferences at scale with an affordable cost.** This capability opens up opportunities for generating large-scale data to support subsequent RLHF (Reinforcement Learning from Human Feedback) and DPO (Direct Preference Optimization) training processes. **This aspect also reflects the "wildness" of our approach, emphasizing its applicability to real-world, complex scenarios.** --- ## Thank You! Thank you again for your thoughtful and valuable comments. Your feedback has provided us with an opportunity to further refine our work, and we are confident that these revisions have strengthened our submission. Please feel free to reach out if you have any further questions or need additional clarifications. We kindly hope that you would please consider raising the score for our submission if you feel that we have adequately addressed your concerns. We would greatly appreciate it, and we are committed to incorporating these responses into the final version of the paper.

Reviewer Response

A1: I acknowledge the first novelty. But for the remaining two novelties, there are numerous works that leverage explanations and checklists to enhance the LLM judge [1][2]. [1] CheckEval: Robust Evaluation Framework using Large Language Model via Checklist [2] TIGERScore: Towards Building Explainable Metric for All Text Generation Tasks --- A2: Thanks for your clarifications. I am looking forward to more fine-grained aspects. --- A3: Thanks for your valuable experimental findings. --- Thanks again for your kind clarifications. I am willing to keep my positive score.

Response to Reviewer LYdL

Thank you for recognizing the value of our work, particularly its real-world evaluation, robust metrics, and dynamic updates. We greatly appreciate your feedback, and below we provide detailed responses to your comments and suggestions. --- ### Cost Comparison Thank you for your suggestion to provide a more detailed cost comparison between WILDBENCH and previous evaluation methods. Using the official codebase from AlpacaEval and ArenaHard, the evaluation cost is approximately \$10 and \$40, respectively. In contrast, WildBench's codebase supports batch mode API evaluation, resulting in a more efficient cost of around \$13-\$17. --- ### Lite Version with Consistency Check Thank you for the suggestion regarding a more cost-efficient version of our benchmark. We have been developing a lite version using 25% of the hardest subset. The selection criteria leverage existing evaluation results from over 70 models to identify examples with lower average scores and higher variance, ensuring the subset remains representative. For this new subset, we are considering using o1-mini for evaluation, which costs around \$5-\$7 per evaluation, while maintaining high correlation with the full evaluation and accurately reflecting differences across models.  We plan to host a new leaderboard with this subset soon, encouraging broader use of our evaluation framework while making it more accessible and affordable. We will continue to provide support for the evaluation of any models that are openly accessible. --- ### Thank You! We sincerely appreciate your time and effort in providing these insightful suggestions. Your feedback has been instrumental in improving both the clarity and practical aspects of our benchmark. We value the opportunity to address your concerns and are confident that the updates we have made strengthen our submission. If there are any further questions or points needing clarification, we would be more than happy to address them.