IDGen: Item Discrimination Induced Prompt Generation for LLM Evaluation

Supplement1 to the rebuttal

**Q:** *In section 2.3 - CoT Check for Mathematical Questions: “we use Hunyuan to assess the reasonableness of the question, which successfully identifies the unreasonableness of the problem and corrects it based on the assessment process.”- How can it be ensured that the model successfully identifies the unreasonableness? Provide a theoretical/experimental study.* **A:** For mathematical questions, we indeed cannot guarantee that the generated questions can always be usable. However, through our proposed inspection mechanism, we can greatly eliminate the problems of **conceptual errors, logical contradictions, violations of common sense, missing conditions, and unsolvable questions**. "The model successfully identifies the unreasonableness" we methion here is **an explanation of the case study in Figure 2**. In Figure 2, we identified the unreasonable part of the question based on the designed CoT, indicating that the question is unusable and needs to be discarded or modified. For mathematical questions, we apply the "instruction gradient" to generalize the questions. **To check the usability of the generated questions**, we design a set of question-checking mechanisms. On the one hand, we design **a set of CoT logic**, starting from the concept, judging the logicality among different parts, evaluating the solvability of the question, and finally checking the question and steps, **gradually guiding the LLM to think about the usability of the question**. On the other hand, we conduct **multi-turn iterative checks through two different LLMs** to ensure the usability of the generated questions as much as possible. Specifically, **the two LLMs independently judge the usability** of the question through the CoT logic, but when one LLM judges the question as unusable or both LLMs judge the question as unusable, the question needs to be modified according to the judgment logic given by the LLM considered unusable (when both LLMs consider the question unusable, we designate one LLM's logic for modification). The modified question is then iteratively checked again. **Only when both LLMs consider the question usable**, the question is considered usable in the production process, and the iterative inspection of the question ends. If the maximum number of iterations is reached and there is still an LLM that judges the question as unusable, the question will be discarded. **Q:** *In section 2.4 - Acquiring reference answers: lines 133-136, are the answers scored by human participants?* **A:** In the selection of reference answers for general text questions, we use the **Hunyuan(Hunyuan-standard)** to score the responses. Using the LLM's response to the instructions as reference answers is **relatively common in the data generation field**. For example, in the Alpaca dataset[1], GPT-3.5 (text-davinci-003) is used to provide responses to questions as reference answers, and in the *Instruction tuning with gpt4 work*[2], GPT-4 is used to answer Chinese questions and serve as reference answers. Despite this, we **hope to improve the quality of reference answers** as much as possible. Inspired by [3], for general text questions, we also **provide seven evaluation criteria**: Safety (0-30 points), Correctness (0-10 points), Relevance (0-10 points), Comprehensiveness (0-10 points), Readability (0-20 points), Richness (0-10 points), and Humanization (0-10 points). We are more inclined to believe that responses with higher scores should have higher quality. We call multiple LLMs, including Hunyuan, GPT-4, GPT4-Turbo, Wenxin 4, and Qwen, to respond to the instructions, and then use Hunyuan to score these responses, selecting the highest-scoring response as the reference answer. We further **involve humans in checking the usability of the answers**. We select 150 generated general text questions and obtain reference answers in the aforementioned manner. We organize evaluators to score the selected reference answers according to the evaluation criteria in Table 1 of the paper. We remove 15 questions that none of the models answer correctly (the questions might be too difficult, all models answer incorrectly, and the answer selection is not meaningful). The results show that the usability rate of reference answers reaches **84.7%**, which is **higher than the highest correct rate of alternative reference answers**, wenxin4 (78.8%). This indicates that the answer selection criteria can ensure the usability of the answers. [1] Taori, Rohan, et al. "Stanford alpaca: An instruction-following llama model." (2023): 6. [2] Peng, Baolin, et al. "Instruction tuning with gpt-4." arXiv preprint arXiv:2304.03277 (2023). [3] Liu, Yilun, et al. "Automatic instruction optimization for open-source llm instruction tuning." arXiv preprint arXiv:2311.13246 (2023).

Supplement2 to the rebuttal

**Q:** *In section 2.4 - Acquiring reference answers: line 140, What is meant by a“collective voting mechanism”? Please explain clearly.* **A:** Thank you for raising this issue, and we appreciate the opportunity to provide a more detailed explanation. For the answers to mathematical questions, we hope to select high-quality responses as reference answers as much as possible. However, it is difficult to design a scoring standard that conforms to human preference for mathematical question responses like general text questions. Related work [1] studies **the theoretical basis of the collective voting mechanism** and discusses the impact of different voting methods on social welfare. Inspired by this, we introduce a "collective voting mechanism" to select reference answers by comparing and voting among multiple responses. We provide multiple **anonymous responses** to the voting LLMs simultaneously, and let the voting LLMs choose the best response they think. We provide multiple voting LLMs, and each voting LLM casts its vote for a response, which means that the voting LLM thinks this response is the best. The response with **the highest number of votes** is used as the reference answer. If there is a tie, we randomly select a response as the reference answer and mark the question. Despite our efforts to enhance the usability of reference answers, there **may still be instances where the selected reference answer is incorrect**. To **further improve the accuracy of reference answers** for mathmatical questions, we hire mathematics experts to check and correct the reference answers of the questions. The results of the manual review are used as the final reference answers. [1] Sen, Amartya. Collective choice and social welfare. Harvard University Press, 2018. **Q:** *In section 2.5 - lines 148-149, what are “label discrimination indexes”? a. In line 149, “the prompt includes four features”- How did you select these features? Provide some analysis. b. In lines 162-164, How did you select the threshold values? (e.g.,“Low”means less than or equal to 0.1,“High” means values greater than 0.25, etc.). c. In line 168, “discrimination level label ranging from 0-3”- Is this range acquired by observations? Or have you performed some analyses on the score expressions?* **A:** Thank you for your question, the label discrimination indexes are the labels mapped from the discrimination indexes you mentioned in the 'b' question, we will improve the presentation here. a. The four features we select are included in each sample: question, its corresponding category, mean length of this category, and length ratio. These features are important and provide meaningful reference for understanding the discrimination of the questions. **Question:** The question is the most direct and key feature. The model needs to understand the question itself. Without the question, it is impossible to determine the type of information provided. **Category:** The discrimination of questions in different categories is usually different. For example, questions in the mathematics category may have different discrimination levels compared to those in the entertainment category. Category information helps us assign appropriate discrimination levels to questions. **Mean length of the category:** Considering the difficulty levels across different categories, the average length of questions within a category can indicate the complexity of the questions in that category. Generally, categories with longer answers may involve more complex questions, while categories with shorter answers may involve simpler questions. Therefore, by comparing the average lengths of different categories, we can gain a rough understanding of the question's difficulty, which serves as an important reference for its discrimination. **Length ratio:** From the perspective of varying difficulties across categories, the length ratio can help us understand the complexity of the question compared to the average question in its category. A higher length ratio may mean higher difficulty, and a lower length ratio may mean lower difficulty. By analyzing the length ratio, we can better understand the relative ranking of the difficulty and discrimination of the question in its category. b. The threshold here is estimated based on the distribution of **100,000-level evaluation data**. c. 0-3 are the four levels we map discrimination indexes to, but this level division is **not unique**, it is just for the convenience of observing data with different discrimination. We can also divide it into two levels, etc.

Supplement3 to the rebuttal

**Q:** *In equation 4, what does the “score”mean? Is it the evaluation score that is depicted in Table 1? a. If you are using the same“score”to calculate the difficulty score and the discrimination indexes, does that mean a question is more difficult if a question is more discriminative?* **A:** Thank you for your question. Yes, the score here refers to the score in Table 1. a. It is the same score, but we believe that the difficulty score can only serve as a reference for discriminability. A high difficulty score for a question does not necessarily mean that it is more discriminative. **For example**, for a question with a max score of 3, if the **evaluation scores are both 0 and 0**, according to the formula, its difficulty score is 3, and the discrimination score is 0, meaning that the question is very difficult, and the LLMs cannot answer it correctly, so **the question is not discriminative**. However, if **the evaluation scores are 0 and 3**, we can calculate that its difficulty score is 1.5, and the discrimination score is 1, indicating that **the question can effectively distinguish the level of LLMs**. **Q:** *While this proposed method is understood to work on general text questions fairly well, mathematical questions are the weakest part of this study.* **A:** Thank you for your affirmation of our work on general text questions. For mathematical questions, on the one hand, providing prompts to large models to generate new questions can **easily lead to unusable questions**, which is difficult to handle; on the other hand, **generating discriminative mathematical questions is also a significant challenge**. We have also done **a lot of work and contributions** on mathematical questions, which are explained here. 1.We have proposed some **data generalization strategies** that can **effectively improve the discrimination and difficulty of questions**. In the "Instruction Gradient", for mathematical questions, we propose 8 generalization strategies to guide data generalization. Experiments have found that the questions generated by these strategies can effectively distinguish the capabilities of existing LLMs. 2.In practical evaluation scenarios, the difficulty of generalizing mathematical questions lies in **the usability of the generated questions**. This paper focuses on solving the problem of low usability of generalized mathematical questions and **designs a usability checking mechanism for mathematical questions**: On the one hand, we have designed **a set of CoT** for checking the usability of mathematical questions. This scheme can guide the LLM to check the usability of mathematical questions from the perspectives of concept, logical relationship, problem solvability, and condition completeness, greatly eliminating the problems of conceptual errors, logical contradictions, violations of common sense, missing conditions, and unsolvable questions. On the other hand, we can **effectively modify or discard unusable generated data through multi-model multi-round iterative checks**. Specifically, we use two different LLMs(refered to Hunyuan-standard and Hunyuan-pro in our paper) to judge the usability of the question based on the above CoT method. For unusable questions, we modify the question according to the judgment of CoT and **iterate the check** again until both LLMs judge the question as usable or reach the maximum number of iterations, and the question is also retained or discarded respectively. Through this mechanism, the generated mathematical questions have satisfactory usability. 3.The **open discrimination estimation model and difficulty estimation model** can quickly **judge the quality of mathematical questions**. In the process of training the discrimination estimation model and difficulty estimation model, we **introduce a large number of mathematical questions** with difficulty and discrimination annotations. The obtained models can effectively and quickly judge the discrimination and difficulty of mathematical questions. We make the models public to facilitate community research or use. 4.We open **a batch of mathematical questions generated by LLMs**(Specially refering to Hunyuan-standard and Hunyuan-pro in our paper) with **accurate reference answers**. We used the Hunyuan large model to generate mathematical questions with **high discrimination**, including **32 types of questions** including calculus, function properties, and arithmetic operations. In addition, we hire **experts in the field of mathematics to check and correct the reference answers** of the mathematical questions to ensure the accuracy of the open mathematical question reference answers.