Official Comment by Authors (Part II)
- Q3: *How much does the new KoLA benchmark differ from existing LLM benchmark? There are so many available benchmark published, each focusing on one/multiple capabilities of the model. Why do we need KoLA? What if we combined existing ones?*
We believe KoLA is unique in two key aspects. (1) As described in Section 1, rather than assessing the breadth of model capabilities like most existing benchmarks (such as MMLU), KoLA focuses more on analyzing the deep interconnections of model knowledge capabilities through carefully-designed multi-level testing. (2) Besides combining some existing tasks, KoLA maintains a unique evolving mechanism, aimed at minimizing data leakage. Recent studies [4,5,6] have widely revealed a serious issue of 'test set contamination' in the evaluation of many LLMs with existing benchmarks, where the results may not genuinely reflect the actual capabilities of the models. Therefore, we believe KoLA holds a distinctive value in the current landscape.
Ref:
[4] Wei T, Zhao L, Zhang L, et al. Skywork: A More Open Bilingual Foundation Model[J]. arXiv preprint arXiv:2310.19341, 2023.
[5] Yang S, Chiang W L, Zheng L, et al. Rethinking Benchmark and Contamination for Language Models with Rephrased Samples[J]. arXiv preprint arXiv:2311.04850, 2023.
[6] Zhou K, Zhu Y, Chen Z, et al. Don't Make Your LLM an Evaluation Benchmark Cheater[J]. arXiv preprint arXiv:2311.01964, 2023.
- Q4: *It is not clear the contribution/motivation with standardized overall scoring. As the score would depend on the evaluated models, it will change a lot as more evaluated models would be added in. Also why this is better than simple ranking?*
Our considerations about employing standardized scoring are as follows: Standardized scores have become the mainstream choice in fields like educational assessment and intelligence testing [7], with systems like the SAT, GRE, and Wechsler Intelligence Scales (IQ Test) adopting them. Compared to absolute values, using standard scores in the multi-task evaluation of LLMs offers the following benefits:
1. Standardized scores enable fair comparisons across different seasons and datasets. The varying difficulty of tasks and the different sensitivities of metrics mean that the absolute numerical performance of models is incomparable. For example, if there is a dataset where models typically perform at 80, and another where the usual performance is 40, directly comparing a model's performance scores on these two datasets cannot reflect whether the model is better at one task over the other. However, comparing standardized scores derived from relative performance can yield conclusive insights.
2. For overall rankings, standardized scores can negate the bias caused by specific tasks. For instance, without standardized scoring, ChatGLM (6B) would rank much higher in overall rankings solely due to its high performance in the 2-2 task, surpassing its improved version ChatGLM2-32k.
We acknowledge that the inclusion of new models indeed results in variations in the overall standardized scores. We believe that considering the benefits mentioned above, such variations are acceptable. One of the objectives of KoLA is to provide a reference for users in selecting models. We trust that if new models demonstrate superior capabilities, the relative downscaling of other models' evaluations is beneficial for users.
Ref:
[7] Haladyna T M, Nolen S B, Haas N S. Raising standardized achievement test scores and the origins of test score pollution[J]. Educational Researcher, 1991, 20(5): 2-7.
**[Final Remark]** Thank you again for reviewing our paper and for the insightful discussions. We hope that our response and clarification have addressed your questions and concerns. Please feel invited to engage with us if you have more questions.