Rebuttal by authors
Thank you for taking the time to review our paper. Our responses are shown below.
---
**W1. Arbitrariness in creating the list of 12 fine-grained skills**
We acknowledge that some arbitrariness exists in defining the 12 fine-grained skills. To establish the 4 abilities and 12 fine-grained skills, we drew upon Rogers et al's (2021) categorization and modified it—for example, changing 'Retrieval' to 'Background Knowledge'—to suit the evaluation of language model alignment. We suggest that future work could refine this categorization, possibly by adding or merging specific skills based on distinct use cases. In this case, FLASK could serve as a foundational framework for fine-grained evaluation. Regarding the specific concern about the arbitrary nature of some skills, we provide the following responses:
1. *Could "readability" be singled out as an "essential skill"?*
We agree that skills such as Readability or Comprehension could be apply all instances. However, our goal of the fine-grained evaluation based on skill sets is to focus on skills that are *truly* essential for the instance. For example, logical problems might not need the skill of readability as much as writing tasks such as “Write a template for a First-Person LinkedIn profile summary.”. To achieve a balance, we found that annotating the top three (K=3) relevant skills per instance was optimal. This approach prevented us from labeling every instance with universal skills like readability while ensuring that instances truly needing those skills were accurately identified.
2. *Could metacognition be labeled in a model-independent manner?*
We acknowledge that the concept of metacognition could be dependent on the specific model being evaluated. However, since language models are evaluated by text-based LLMs or humans in this work, our reference standards are aligned with the capabilities of GPT-4. Therefore, instances that require other modalities (ex) Do you like the taste of pizza?) or predicting future events (ex) If bitcoin has gone up in value over the last twenty years, what do we know will happen in the next twenty years? Will a 100 trillion parameter deep learning model be trained before 2026?) are mainly annotated to include metacognition.
We have added the details regarding the specific skills in the revised paper. (Appendix E)
Reference:
Rogers et al (2021), QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
---
**W2. Missing domains and multiple domains**
1. Missing domains
We divided the domains into 10 domains and 38 sub-domains as shown in Table 6 in the Appendix. Domains such as laws and finance are classified as Social Science domains according to the table and the travel domain should be classified as Culture considering the similarity with the sub-domains of the Culture domain. During domain annotation, we also provide the sub-domain information in the prompt to specify the boundary between various domains.
2. Multiple domains
We allow multiple domain (up to two) annotations during the domain annotation process by specifying in the annotation instruction that two domains may be labeled when it is hard to annotate only a single domain.
---
**W3. Comparison of open and closed source LLMs**
We acknowledge that it is hard to draw conclusions for closed-source LLMs since the model details are not provided. We also agree with your point that the performance gap between open-sourced and closed-source LLMs might be due to the differences in model size rather than specific training techniques. We have revised the paper by deleting hypothetical interpretations regarding the comparison between open-sourced and closed-source LLMs to avoid misinterpretations.
---
**W4. English presentation**
We have updated the paper to enhance the English expressions. Thank you for your suggestion.
---
**Q1. Instance selection criteria for each dataset consisting of FLASK**
For the construction of the FLASK evaluation set, we prioritized diversity to prevent bias towards any particular distribution. In single-task datasets, we limited the number of instances to a maximum of 20. For multi-task datasets such as Self-Instruct and WizardLM, we observed the inherent diversity between instances and did not impose this restriction. For multi-task datasets such as MMLU and BBH, we sampled a single instance per sub-dataset since there was a significant overlap of required skills, domain, and difficulty between the same sub-dataset. The full distribution of each dataset consisting of FLASK is presented in Table 12 in the Appendix.
---
**Q2. Plan for open-sourcing the FLASK dataset**
Yes, we are planning to open-source the dataset, the metadata annotation, the evaluation pipeline (code implementation), and the interactive visualization demo page for analysis.