Small Language Model as Data Prospector for Large Language Model

The quality of instruction data directly affects the performance of\nfine-tuned Large Language Models (LLMs). Previously, \\cite{li2023one} proposed\n\\texttt{NUGGETS}, which identifies and selects high-quality quality data from a\nlarge dataset by identifying those individual instruction examples that can\nsignificantly improve the performance of different tasks after being learnt as\none-shot instances. In this work, we propose \\texttt{SuperNUGGETS}, an improved\nvariant of \\texttt{NUGGETS} optimised for efficiency and performance. Our\n\\texttt{SuperNUGGETS} uses a small language model (SLM) instead of a large\nlanguage model (LLM) to filter the data for outstanding one-shot instances and\nrefines the predefined set of tests. The experimental results show that the\nperformance of \\texttt{SuperNUGGETS} only decreases by 1-2% compared to\n\\texttt{NUGGETS}, but the efficiency can be increased by a factor of 58.\nCompared to the original \\texttt{NUGGETS}, our \\texttt{SuperNUGGETS} has a\nhigher utility value due to the significantly lower resource consumption.\n