Cookbook: A framework for improving LLM generative abilities via programmatic data generating templates

Summary

This paper introduces an automatic framework called Cookbook to construct instruction tuning data without the involvement of human-written data and LLM-generated data. Cookbook generates instruction-tuning data according to programmatic templates of pattern-based rules, and applies the rules to a random token space. To improve performance on multiple tasks, the authors also propose a mixing algorithm over Cookbook templates called Cookbook-Mix. Experiments show that Llama-2 and Mistral finetuned on Cookbook-Mix generated data outperform the base models on the GPT4All benchmark. Analysis shows that the rules taught by templates are crucial to model performance.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

1. The idea of constructing instruction datasets via pattern-based rules over random tokens is interesting and courageous. It could be a great starting point and enlighten future works. 2. The paper is well-written. It clearly describes how the templates are constructed and mixed, and how the template alignment scorer works. The analysis section is well-organized and addresses several potential concerns.

Reasons to reject

1. All the rules designed for tasks in the main content of the paper are based on token overlapping. This may limit the scope of tasks to which this method can be applied, and may teach models to learn shortcuts. For example, the choice that overlaps more with the question is not necessarily the correct answer. The authors acknowledge the restriction and add the task of poetry generation in the appendix, but the template needs more human effort like designing the rhyme_dict, and is unlikely to be generated automatically. 2. The experiments could be further supplemented to make the claims more convincing. a) The framework could be evaluated on more benchmarks besides GPT4ALL, like Open LLM Leaderboard and BigBench-Hard. More instruction-tuned Llama-2 and Mistral models report their performances on these benchmarks. b) The instructed versions of Llama-2 and Mistral (Llama-2-chat and Mistral-Instruct) could serve as baselines. c) For the single-task evaluation, the performance of the 7B models could also be reported.

Summary

The paper introduces Cookbook, a framework designed to generate instruction datasets that enhance large language model (LLM) performance without relying on human or LLM-generated samples. Instead, Cookbook programmatically creates training data using simple patterns over random tokens via Python functions. It generates training data to help the model learn explicit rule-based patterns for specific tasks. The authors also propose a method to mix data from different templates to improve performance across various tasks simultaneously.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

**1. Interesting method**: It is very interesting to use code to help us automate the training data generation process. It is a completely novel paradigm than the pure natural language-based methods.

Reasons to reject

**1. Limited application scenarios**: Based on the examples in Figure 2 and my analysis, the program appears to generate very preliminary span-level or lexical-level training data. It is likely due to the ease of using Python string operations for synthesis. However, this approach seems less effective for generating more semantically complex QA questions or broader user instructions, as Python code alone may struggle to manipulate question semantics effectively. Also, the authors focus their model testing exclusively on QA-related tasks, rather than exploring a wider range of more general tasks that would better demonstrate the framework's broader applicability. **2. Lack of more in-depth evaluation**: The authors use code to generate training data but overlook other baselines that also facilitate QA data generation. At least, the authors could conduct experiments using Self-Instruct methods to generate QA training data and verify whether the proposed approach truly surpasses Self-Instruct. In my view, particularly for QA tasks, Self-Instruct-style methods appear better suited for generating a broad range of QA questions, irrespective of domain, text length, or format. Moreover, the baselines currently compared to Cookbook weren't specifically trained on QA-style data, potentially contributing to their lower performance relative to Cookbook. **3. Weird performance linear assumption**: The authors introduce a method to control template proportions in training data based on the assumption that the accuracy of a model trained on a weighted mixture of templates is equivalent to the weighted average accuracy of models individually trained on each template. However, this assumption may be flawed because models specifically trained on one template (e.g., A) tend to perform better on corresponding template-based QA tasks than models trained on a more diverse mixture of templates (e.g., A, B, C, D). In this case, B, C, and D would be out-of-distribution data for the A-based test set. The authors need to conduct experiments to validate this assumption before making claims about its feasibility.

Summary

This paper presents an approach, generating instruction dataset prgramatically, in stead of relying on human or LLM based instrcution dataset. Three downstream tasks were selected, where the process of generating datasets were detailed. The paper also indicated that the programmatic data generation contributes to privacy by operating over a random token space, which avoids privacy and legal concerns as it does not involve the use of real user data or violate the terms of service of LLM providers. The evaluation is carried using accuracy (single and mixed - multitaks setup). The paper is easy to read. However, the way the datasets are selected prgramatically, for example for the QA, which part of the document and what kind of question would be included is not clear. It is mentioned that LLMs are involved in the process but I could not understad the process. Moreover it is not clear or checked how the approach solve the privacy issue.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

- Programatic instruction data generation - Evaluation approaches in place

Reasons to reject

- The theoretical appraoch of generating the dataset is sound, but practically, how can we generate (or select ) the dataset from the source is not clear - The distiniction from the LLM based instruction dataset generation is not clear - A lot of important content are presented in the Appendixes

Questions to authors

You have a lot of concept in the appendix which makes the paper hard to understand.

Summary

This paper introduces COOKBOOK, a new framework designed to boost the generative abilities of LLMs with programmatically generated training data. Unlike recent methods that rely on costly human-curated or LLM-distilled datasets, COOKBOOK uses Python functions to create data templates that could generate random token samples as the instruction-tuning instances. These templates teach models specific pattern-based rules, making the data generation process scalable and free from privacy concerns. The authors show that models fine-tuned with COOKBOOK could significantly improve their performance on various NLP tasks. Besides, the COOKBOOK-MIX algorithm can select data from multiple templates to further enhance performance. Experimentally, the results are impressive. Models fine-tuned on COOKBOOK beat other instruction-tuned LLMs on several benchmarks. The authors also conduct in-depth analyses to show that why templates based on random token sequences can improve model performance.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

1. Using templates to generate instruction-tuning data with random token sequences is a novel idea for me. It appears that LLMs have sufficient natural language understanding to learn instruction-following abilities even from pattern-based, unreadable pseudo-data. 2. The authors conducted extensive experiments on different LLMs across various benchmarks, and the results are impressive, strongly supporting their claims. 3. Additionally, the authors provided empirical analyses to explain why these template and random token sequence-based data still work effectively.

Reasons to reject

1. The baselines (instruction-tuning datasets) compared by the authors seem to focus primarily on open-ended conversations, while the tests mainly involve traditional NLP tasks. Including more instruction-tuning datasets on these NLP tasks like FLAN could make the experiments more convincing. 2. The experimental descriptions in the paper are sometimes too brief, making it hard to follow. For instance, the sizes of the different datasets used (both baseline and COOKBOOK) and various training details are either omitted or scattered across the Appendix. 3. I find the design of the QA-related templates unclear and not well-motivated. For example, for Document QA, why is the answer set to the k tokens before and after the question? And why is k set to 3 (Appendix B.3)?

Questions to authors

I have a question regarding data size. Intuitively, if too much unreadable data is introduced during the SFT stage, it might potentially degrade the model's natural language understanding capabilities. Did the authors conduct experiments on this? Although the proposed method can generate unlimited data scalably, could introducing too much of this data gradually impair the model's overall generative ability?