Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples

Summary

The paper presents a programmable natural language reasoning dataset, PrOntoQA-OOD, specifically designed for testing the out-of-distribution generalization abilities of machine learning models. 4 LLMs, FLAN-T5, LLaMA, GPT-3.5, and PaLM, are tested on this dataset for their ability to generalize to different deduction rules, proof width/depth, and rule compositions.

Strengths

- The dataset construction is solid and takes into consideration various deduction rules. - The experimentation is thorough and insightful. - The paper is well written.

Weaknesses

- (Major) [1] presents a benchmark for theorem proving that is programmable and tests generalization across six dimensions. These dimensions include different deduction rules and proof width/depth, which this paper also discusses. It's a very relevant paper and the failure to discuss it as related work in context seems like a major oversight in scholarship. But this is quite easy to fix. - (Minor) Footnote 2 says the deduction rules used in this paper have minor differences with Gentzen's natural deduction, yet there are 3 rules omitted from it. This seems more than minor and I fail to see how natural deduction's completeness or soundness claims can translate to this paper's, without substantiation. - (Minor) Figure 3 notes GPT 3.5 as 175B* with an asterisk indicating low confidence in its size. I feel like it's best to omit the 175B size claim altogether if the authors are unsure, to avoid misleading. - (Major) For the figures, OOD generalization performances are shown in $\Delta$ proof accuracies w.r.t. the ID ones. This is hard for me to get an intuitive understanding of the different models used, particularly for FLAN-T5: FLAN-T5 often has trivial (close to 0) proof accuracies for ID and its OOD performance $\Delta$ being 0 is an indication of triviality instead of good generalization. Showing the absolute OOD proof accuracies will solve this problem. - (Minor) For Figure 5 and several others, there's an example of (predicted answer, expected answer), without the question. I had to use my inductive reasoning to figure out exactly what the question was. Please fix this. - (Minor) Line 17 seems to cast theorem proving and medical diagnosis as purely deductive reasoning problems. Both tasks require inductive reasoning among other modes. So this claim is not rigorous. Overall, there are problems but all seem fixable. I'm happy to change my rating if the authors can amend the paper to be more satisfactory. [1] Wu, Y., Jiang, A.Q., Ba, J. and Grosse, R., 2020. Int: An inequality benchmark for evaluating generalization in theorem proving. arXiv preprint arXiv:2007.02924.

Questions

More contextualization with prior work should be good.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

More discussion on limitation of this paper's approach for evaluating LLMs reasoning would certainly improve the paper.

Summary

This submission proposes a new dataset PRONTOQA-OOD to systematically study the general deductive reasoning capabilities of large language models (LLMs). Some findings are obtained, e.g., LLMs exhibit mixed generalization to unseen deduction rules and more robust generalization to longer, wider, and compositional proofs than previously suggested.

Strengths

Summary: (1) A new dataset for evaluating the deductive reasoning capabilities of LLMs is proposed. (2) Some new findings on the deductive reasoning capabilities of LLMs are presented. (3) The submission is well written and easy to follow. Soundness: The construction method for the proposed dataset PRONTOQA-OOD is sound, using natural language templates from deduction rules and fictional names for all concepts, where fictional names rather than real names are exploited to prevent LLMs from reusing knowledge from pretraining. The findings are drawn from careful experimental designs and adequately supported with empirical evidence. Presentation: The submission is well written and organized, with sufficient technical details, easy to follow. Contribution: Evaluating the deductive reasoning capabilities of LLMs is important in the field of artificial general intelligence. The submission pushes forward this study by considering more natural deduction rules and drawing some new findings. The findings are valuable to share with the broader NuerIPS community.

Weaknesses

Similar to existing datasets for evaluating the deductive reasoning capabilities of LLMs, the proposed dataset is built on simple and short templates mimicking deduction rules. The sentences used to evaluate LLMs are different from general natural language sentences. It may not reflect the deductive reasoning capabilities of LLMs over natural language sentences. There are two minor issues on presentation. One issue is on the definition of the same distribution, which has been referred to in several places including Abstract. It is unknown how to determine whether two examples are obtained from the same distribution or from different distributions. The other issue is on the mentions of abbreviations. An abbreviation should be given its full name when it is mentioned for the first time, however, some abbreviations such as OOD do not come with their full names when they appear for the first time.

Questions

How is the same distribution defined? That is, how do we determine whether two examples are obtained from the same distribution?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I do not find limitations that the authors do not address in their methodology or mention in future work.

Summary

In this paper, the authors explore the general deductive reasoning ability of LLMs. They test LLMs on a broad set of deduction rules and measure the ability to generalize to more complex proofs from simpler demonstrations from multiple angles. They construct a programmable dataset that enables control over deduction rules and proof complexity. The experiments are conducted on four LLMs with different sizes and training objectives.

Strengths

Overall, I think the paper is addressing an important research direction that helps better understand the general deductive reasoning ability of LLMs. Several strengths I particularly admire are: 1. The category of rules and studies of models under various settings are systematic and comprehensive. Many takeaways are provided (although I believe some of them can be made clearer). 2. The creation of a new synthetic and programmable reasoning dataset is a valuable contribution. This dataset allows for control over deduction rules and proof complexity, which can be beneficial for future research in this area.

Weaknesses

While the paper presents comprehensive experiments to assess the reasoning capabilities of LLMs, there are aspects that require clarification: 1. The main issue with the writing is that the outcomes of the experiments are not clearly communicated, likely due to a lack of correlation between figures and claims. For example, the paper asserts that "LLMs can generalize to more complex proofs but require explicit demonstrations to produce hypothetical subproofs". It would be beneficial if the authors could explicitly highlight which experiments (and comparisons) substantiate this claim. 2. A critical concern, albeit not too severe, is the absence of a discussion about the underlying mechanisms and the implications or applications of the observations. Various claims are made, but the *reasons* behind these phenomena and *how* we can improve the models aren't elaborated on. For instance, ICL exhibits variable performance under different deduction rules, which complicates the decision of its usage. If the authors could provide explanations, identify the factors causing ICL's performance degradation, and propose potential solutions, it would significantly enhance the paper's value.

Questions

1. In line 183, "we observe that the gap in proof accuracy between the ID and OOD settings is quite small". However, in figure 6, it seems like you observe an accuracy drop of 20% in some experiments. 2. Different from previous works, this paper claims that LLMs can generalize compositionally. Can you further compare your work with them, and analyze why different conclusions are made? 3. I cannot understand the subtitle of section 4.1. Many a more concise and academic title would be better.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Not applicable.

Summary

The authors propose a new dataset ProntoQA-OOD, to evaluate LLMs' reasoning abilities across more complex proofs across multiple aspects such as depth, compositionality, etc. The authors find that LLMs are able to generalize to longer and compositional proofs but usually require demonstrations to get these good accuracies.

Strengths

* The analysis done in the paper is pretty comprehensive and contains some interesting findings that can help future LLM research. * The dataset is an interesting resource for future reasoning-based evaluation of LLMs

Weaknesses

* The main weakness is the lack of meaningful insights from the results, which are mostly contradictory to previous findings. Since the majority of the observations contradict previous works (section 4.2.1, 4.2.2, 4.3), its not sufficient to just state the observation and note its in stark contrast to previous findings. It'd help to take few of these scenarios and deep dive on why the observations are so different. Likely reasons could be the kind of datasets, number of ICL examples, etc. Please add a case study comparing these models across previous datasets and this dataset to make the analysis more interesting. * Another issue with so many surprising results where the authors claim that LLMs are good at a task (which previous papers found to be hard to solve), leads to the question about the specific data distribution being used to test these models. Maybe the data is very basic and in-distribution (i.e., the models already have strong knowledge about these instances). Comparisons with a synthetic dataset such as ProofWriter can help alleviate this issue. * Observations made by the authors in Section 4.2.2 (lines 192-193) that its not always best to provide demonstrations from the same distribution is in contrast to the general claim that LLMs do require in-demonstration examples (contribution 3). More discussion on this point needs to be done to explain when some examples are helpful and when they are harmful. * The work on expanding compostionality is very similar to a prior work [1] that explores OOD compositonality. Consider discussing that in the related works. [1] RobustLR: A Diagnostic Benchmark for Evaluating Logical Robustness of Deductive Reasoners, EMNLP 2022

Questions

See weakness section for suggestions.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Response to authors

Thanks for conducting these experiments quickly and adding more insights. Both the findings about proof depth vs width and addition of redundant distractors are interesting. These points should be highlighted not just in the relevant sections, but also in introduction, and possibly conclusions. I'm happy to increase my scores to reflect my new position about the paper.