We thank the reviewer for the feedback.
# Weaknesses
> 1. **Motivation and problem**
- We focus on the topic of utilizing large language models (LLMs) for evaluating open-ended generation instead of proposing a decoding strategy. The goal is to propose new methods using LLMs for better and fair evaluations that align with human judgments.
- The basic task formulation is like, given answers A and B, the model is required to evaluate which of them is better.
- Determining pairwise comparison is also a key step for InstructGPT training [7].
- To get an overview and basic knowledge of the topic, we kindly refer the reviewer to skim a set of related work on LLM-based evaluation [1,2,3,4,5,6,7], where [1] is a very basic model in this direction.
> 2. **Proposed solution**
- We propose (1) the peer rank (PR) algorithm that takes into account each peer LLM’s pairwise preferences of all answer pairs and outputs a final ranking of model capabilities;
- And (2) peer discussion (PD), where we prompt two LLMs to discuss and try to reach a mutual agreement on preferences of two answers.
> 3. **Results**
- We conduct comprehensive experiments on two benchmark datasets. We find that our approaches achieve significantly higher accuracy and better alignment with human preference.
# Questions
> 1. **Is the aim of your method to help LMs produce better answers to questions than other methods currently achieve?**
- As outlined in the abstract and introduction, our aim is to **introduce improved evaluation methods for large language models (LLMs)** that enhance accuracy and better align with human judgments. Our research opens up possibilities for assessing models that are challenging for humans to compare.
**Finally, we are eager to answer any further questions from the reviewer to help fully understand the intellectual merits of our work.**
# References
1. Fu, Jinlan, et al. "Gptscore: Evaluate as you desire." arXiv preprint arXiv:2302.04166 (2023).
2. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. Gpteval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634, 2023.
3. Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators, 2023a.
4. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
5. Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. Benchmarking foundation models with language-model-as-an-examiner. arXiv preprint arXiv:2306.04181, 2023.
6. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
7. Ouyang, Long, et al. "Training language models to follow instructions with human feedback." Advances in Neural Information Processing Systems 35 (2022): 27730-27744.