Harnessing the Power of Large Language Models for Natural Language to First-Order Logic Translation

Response

Thank you for your comments and we appreciate your positive feedback on our work. Our responses are as follows: ***”The novelty of the proposed approach is not totally clear”*** We thank the reviewer for acknowledging the novelty of our proposed RLHF method. We would like to further highlight two important aspects of our work to clarify the overall novelty: **LogicLLaMA and MALLS are significant contributions to the logic and NLP literature.** NL-FOL translation has been a long-standing challenge in both NLP and the formal logic literature, and it plays a central role in many logic-based AI systems [1-4]. Solving this task could open up a wide range of applications, yet there lacks such a translation model that scales to real-world data, preventing these systems from applying to real-world NLP problems. We believe LogicLLaMA is a significant contribution to the literature (As also acknowledged by reviewer wVqn) as it, for the first time, provides a GPT4-level performance translation model that is cheap, open-source, and can be extended for downstream tasks. **The creation of MALLS dataset is non-trivial; prior work with similar scope is widely appreciated.** This work shares a similar scope as those self-instruct LLM work such as Alpaca [5], Vicuna [6], and LLaVa [7]: most of them use the existing models and training algorithms, and the main contributions are “engineering” work such as dataset creation, prompting techniques, and so on, yet they are highly influential and significant to the community. We note the creation of the MALLS dataset is also one of the main contributions of this work and should not be neglected. Given the importance of NL-FOL translation in many logic-based NLP systems, a high-quality real-world NL-FOL pair dataset is highly valuable to the literature and can open up possibilities for many downstream applications. In this regard, MALLS stands out as it is significantly larger (14x times compared to FOLIO) and more diverse in terms of logical expression and context. **RLHF naming** Sorry for the confusion. Yes, there is indeed no human feedback involved. Technically, it should be RL with FOL verifier feedback, but since the overall algorithm remains the same we keep the same naming convention. We have included clarification for this in the draft. [1] Abzianidze, L. (2017). LangPro: Natural Language Theorem Prover. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 115–120. [2] Bos, J., & Markert, K. (2005). Recognising Textual Entailment with Logical Inference. Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, 628–635. [3] Lu, X., West, P., Zellers, R., Bras, R. L., Bhagavatula, C., & Choi, Y. (2021). NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints (arXiv:2010.12884). [4] Wang, S., Zhong, W., Tang, D., Wei, Z., Fan, Z., Jiang, D., Zhou, M., & Duan, N. (2021). Logic-Driven Context Extension and Data Augmentation for Logical Reasoning of Text (arXiv:2105.03659). [5] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. [6] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng,Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. [7] Liu, Haotian, et al. "Visual instruction tuning." arXiv preprint arXiv:2304.08485 (2023).

Response

Thank you for your comments. Our responses are as follows:   ***”The paper lacks novelty”*** We clarify this point on three aspects: **The iterative correction with RLHF is technically novel.** We emphasize that the iterative correction with RLHF is a novel extension (as also acknowledged by reviewer 8JQp) to the original RLHF setup. Prior work either realized iterative correction via pure prompting [1] with closed-source models such as ChatGPT, or utilized RLHF for other tasks [2] such as instruction-following fine-tuning. Ours is the first method that uses RLHF to fine-tune language models for logical understanding with non-trivial settings, such as FOL rule perturbation and logical equivalence evaluation. **LogicLLaMA and MALLS are significant contributions to the logic and NLP literature.** NL-FOL translation has been a long-standing challenge in both NLP and the formal logic literature, and it plays a central role in many logic-based AI systems [3-6]. Solving this task could open up a wide range of applications, yet there lacks such a translation model that scales to real-world data, preventing these systems from applying to real-world NLP problems. We believe LogicLLaMA is a significant contribution to the literature (As also acknowledged by reviewer wVqn) as it, for the first time, provides a GPT4-level performance translation model that is cheap, open-source, and can be extended for downstream tasks. **The creation of MALLS dataset is non-trivial; prior work with similar scope is widely appreciated.** This work shares a similar scope as those self-instruct LLM work such as Alpaca [7], Vicuna [8], and LLaVa [9]: most of them use the existing models and training algorithms, and the main contributions are those so-called “engineering work”: dataset creation, prompting techniques, and so on, yet they are highly influential and significant to the community. We emphasize the creation of the MALLS dataset is also one of the main contributions of this work and should not be neglected. Given the importance of NL-FOL translation in many logic-based NLP systems, a high-quality real-world NL-FOL pair dataset is highly valuable to the literature (for benchmarking or fine-tuning) and can open up possibilities for many downstream applications. In this regard, MALLS stands out as it is significantly larger (14x times compared to FOLIO) and more diverse in terms of logical expression and context.   ***” The proposed methods do not show significant improvement”*** As stated above, we disagree that one views RLHF as the sole proposed method and the rest two modes as the baselines—they are different modes of a LLaMA model proposed in this work for NL-FOL translation, and they are trained and evaluated on a dataset is non-trivial to collect and process. Nevertheless, we are glad to share that we have resolved a few implementational issues with RLHF and rerun the experiments. We find the updated iterative correction performs significantly better compared to the two other modes, leading to ~10% increase in both FOL BLEU and FOL LE on FOLIO and MALLS (whereas scores for LogicNLI are already saturated). We have updated the RLHF scores in Table 2.   ***” Can your method generate more examples?”*** Yes, and we plan to collect more FOL rules in future work.   ***” How do you assess the coverage or diversity of your dataset? LLM may only generate high-frequency knowledge.”*** We believe the **Pair diversity** paragraph in section 3.3 has sufficiently addressed this concern. During the data collection phase, we implemented several prompting techniques to encourage rule and term diversity. As a result, *“MALLS has a total term vocabulary size of 49394 and the most frequent terms occur less than 2K times (Figure 8 in Appendix B), suggesting a diverse vocabulary distribution.”*. We also show in Table 1, Figure 1, and 7 that MALLS covers a wide range of concepts as well.   ***”While the natural language input generated from LLM is often fluent and grammatically correct, real human inputs can be noisy. How do you address this issue?”*** As stated in Table 1 and Section 2, FOLIO consists of NL-FOL pairs written by humans, and we believe evaluating LogicLLaMA on this benchmark shows how well the model performs with *real human inputs*. As a result, Table 2 suggests that LogicLLaMA performs well on it.

Concerns about the novelty

I agree with the author regarding the value of the MALLS dataset. However, I have some concerns about the contributions: 1. LogicLLaMA: Unlike Alpaca, Cicuna, and LLaVA, which aim to enhance open-domain instruction following ability or multi-modal ability, LogicLLaMA primarily focuses on first-order generation ability. I am concerned that this gain may result from adapting open-domain to a specific domain, potentially harming open-domain ability. To address this concern, could the author provide tests of LogicLLaMA on tasks beyond logic translation? At a minimum, please test the model on logical, reasoning, and math-related datasets, which may benefit from the ability to generate logic expressions. 2. Performance of RLFH correlation: Based on the updated Table 2, the gain of RLHF correlation is still marginal. I agree with the author that the "LogicLLaMA-13B Trans." setting achieves more than a 10% gain compared to "LLaMA2-13B 5-shot", demonstrating that fine-tuning with MALLS can improve performance on these three datasets. However, when evaluating the proposed methods "Corre" and "RLHF Corre", we should compare them with the "trans" setting. Most of the gains are less than 1%, which is marginal. To support my concern, could the author run the fine-tuning experiment multiple times and report the variance of the results? 3. Creation method of MALLS: Does the creation method possess novelty that could benefit the community and generalized to creation of other datasets? I am unable to determine this on my own.

Response

Thank you for your comments. We appreciate your positive feedback on our work! Our responses are as follows: ***”Is it a fully expressive First-order logic language or just the logic program subset?”*** Yes, both MALLS dataset and LogicLLaMA are for general FOL rules. When collecting and validating the FOL rules, we do not enforce specific semantic models such as the Herbrand structure. On the logic expression level, the dataset does have some biases, where it contains more entailment-type FOL rules and fewer rules with rare logical operators such as xor. For measuring logical equivalence, we follow a similar protocol as in FOLIO[1] that converts them to propositional rules and compares the truth table. **Iterative correction presentation; input, output, and loss functions of T1-T4** Thank you for the suggestions. We have included a summary of these items in Appendix C, where we summarize the input, output, and objectives of the four tasks. [1] Han, S., Schoelkopf, H., Zhao, Y., Qi, Z., Riddell, M., Benson, L., Sun, L., Zubova, E., Qiao, Y., Burtell, M., Peng, D., Fan, J., Liu, Y., Wong, B., Sailor, M., Ni, A., Nan, L., Kasai, J., Yu, T., … Radev, D. (2022). FOLIO: Natural Language Reasoning with First-Order Logic (arXiv:2209.00840). arXiv. http://arxiv.org/abs/2209.00840