Author's Response to Reviewer tbcE
Thank you for your valuable and meaningful feedback and for pointing out typos in our work; we updated our manuscript, reflected reviews and fixed typos, and carefully addressed your questions and concerns below.
> **How did you get access to the dataset? How did you ensure the dataset has not been included in LLM? In what years these final exams were given?**
We collected 1,000 NLP exam questions over a decade, from 2013 to 2023. For the integrity and fairness of the final exams, these questions have not been available online, so they cannot be part of other datasets, and there is no risk of training data contamination. Professors and teaching assistants (TAs) contributed new questions to this repository each semester. This extensive set comprises three types of questions: 1) Online-sourced questions, refined by TAs to differentiate them from their original versions, ensuring their uniqueness for final exams, along with a thorough verification of answers. 2) Original questions formulated by professors, drawing from their teaching experience. 3) Original questions developed by TAs. Over 30 TAs have been involved in the completion of this dataset. We have made this clear in the updated version.
> **What were the criteria for data selection?**
In the raw data, many questions include figures and tables in the context. We discard those questions since we cannot easily translate them into text formats like images and tables. We have made this clear in the updated version. Considering that multimodal LLM (like GPT-4V) has been announced after this work, we will add more experiments of multimodal models in our future work.
> **Why not compare dataset size?**
A good benchmark dataset should be able to reveal the differences among LLMs through their performance. According to Table 3, Figures 2 and 3, NLPBench can achieve this with the current amount of questions. We also provided in-depth error analyses in Section 4. Therefore, we did not focus on tweaking the number of questions in each dataset but more on the features (e.g., new types of questions or new evaluation processes).
> **Did you check if NLPBench's math-related problems are not also in SciBench?**
SciBench extracts questions from ten textbooks in the fields of physics, chemistry, and pure math, while the questions in NLPBench are created by professors and TAs. Therefore, there is no overlap between SciBench and NLPBench.
> **How many human annotators are involved?**
The main contributors to the original dataset are at least 30 TAs, each of them completing part of the questions’ or answers’ completion. Furthermore, three human experts are involved in the data processing stage: one for transiting the original format into the text format with a front-end interface and two for checking the results. It took about one week to complete the development of the interface and dataset transformation. Two of them were further involved in the free-response answer evaluation and error analysis stage. Evaluating each answer needs ~5 minutes per annotator, and it took about 13 hours per annotator to finish the evaluation.