Thanks
Thanks for your feedback! Let us know if your have followup questions.
1. **The idea is not very novel, and similar ideas have already been explored. For example, [1] also constructed a dataset using similar constraints for instruction fine-tuning.**
* Thanks for bringing up this concurrent work, which we have cited and discussed in the revised paper!
* It focuses on turning constraints into NL instructions and fine-tuning smaller language models such as T5-11B, and the constraints themselves are fairly simple and limited (only 5 types). In contrast, Collie focuses on making constrained text generation more flexible, customizable, and harder, challenging even SoTA LLMs like GPT-4.
* We believe these two efforts are orthogonal and can be complementary --- we can use these harder and more flexible constraints from Collie-v1 and other Collie constructed datasets to diagnose and analyze LMs (as shown in the paper), and improve the instruction tuning of LMs.
2. **The dataset may not be very useful. Specifically, because the rules are too vague/arbitrary, the extracted ground truth is not useful for the evaluation process: the authors only use them for comparing fluency. In addition, since the rules can be arbitrarily designed, this compiled dataset does not hold much value, because many similar datasets can be compiled with different engineering details.**
* The extracted ground truth text is not only useful for comparing fluency, but also to ensure the extracted constraint values are solvable using natural text. We believe it is actually important to evaluate constraint satisfaction using the constraints instead of the extracted text, as there could be very diverse generations that all satisfy the same constraints.
* As stated in the paper, our main contribution is twofold: the Collie framework for constructing constrained text generation tasks, and the Collie-v1 dataset using a concrete and preliminary set of constraint types. The facts that “rules can be arbitrarily designed” and “many similar datasets can be compiled with different engineering details” are exactly the merit of our proposed Collie framework --- while previous datasets feature a fixed set of simple constraints and ad-hoc pipelines for data collection, the Collie framework allows automatic and scalable construction of arbitrary grammar-expressible constraints.
3. **In the limitations, the authors mentioned potential problems with filtering and processing functions. Could the authors elaborate on what issues might exist for the extraction process?**
* Every text corpus has artifacts (e.g., tables, footnotes, references, headings that might not be natural sentences or paragraphs). We design filtering and processing functions to patch or eliminate passages with these artifacts. However, it is impossible to design a filtering system that is perfect, so there could be downstream reference texts or constraints that look unnatural. This is more of a fundamental limitation with any approach that leverages text in the wild, and not a limitation specific to Collie or CTG construction. We have revised the limitation part to better explain this, thanks!