Library

WILT: A Multi-Turn, Memorization-Robust Inductive Logic Benchmark for LLMs

While large language models have shown impressive capabilities across a wide range of domains, they still encounter significant challenges in reasoning tasks that require gathering evidence over multiple turns and drawing logical conclusions. These challenges present significant obstacles for LLM chat user interfaces, which rely on multi-turn interactions to facilitate effective collaboration. This limitation leads to real-world issues; for example, service chatbots must gather necessary information from customers over multiple turns to diagnose and resolve problems effectively. Despite the multi-turn nature of many real-world LLM use cases, most existing benchmarks rely on carefully curated single-turn tests, which often blur the line between memorization and genuine reasoning. To address this, we introduce the Wason Inductive Logic Test (WILT), a simple yet challenging multi-turn reasoning benchmark designed to resist memorization. WILT is inspired by the Wason 2-4-6 task, where participants must infer a boolean function involving three variables (e.g., $x<y<z$) by proposing test cases (such as $(2, 4, 6)$). In WILT, each test starts from a clean slate, with only the initial instructions provided, preventing models from relying on pre-learned responses. Over several turns, models must interact with the environment by suggesting test cases to narrow the possible hypotheses and ultimately infer the hidden function based on the outcomes. Our findings reveal that LLMs struggle with this task, exhibiting distinct strengths and weaknesses: some are better at narrowing down the hypothesis space by proposing valuable test cases, while others are more adept at deducing the hidden function from observed cases. Despite these variations, the best-performing model achieves only 28% accuracy, highlighting a significant gap in LLM performance on complex multi-turn reasoning tasks.

cs.AIComputer Science
View graphSource

Paper

Similar papers

FALSIFYBENCH: Evaluating Inductive Reasoning in LLMs with Rule Discovery GamesLeonardo Bertolazzi, Katya Tentori, Raffaella Bernardi2026 · 0 citations In Library
Multi-Turn Puzzles: Evaluating Interactive Reasoning and Strategic Dialogue in LLMsKartikeya Badola, Jonathan Simon, Arian Hosseini et al.2025 · arXiv.org · 8 citations In Library
TurnBench-MS: A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language ModelsYiran Zhang, Mo Wang, Xiaoyang Li et al.2025 · arXiv (Cornell University) · 9 citations In Library
InductionBench: LLMs Fail in the Simplest Complexity ClassWenyue Hua, Tyler Wong, Sun Fei et al.2025 · Annual Meeting of the Association for Computational Linguistics · 6 citations In Library
RiddleBench: A New Generative Reasoning Benchmark for LLMsDeepon Halder, Alan Saji, Thanmay Jayakumar et al.2026 In Library
LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language ModelsMihir Parmar, Nisarg Patel, Neeraj Varshney et al.2024 · arXiv (Cornell University) · 88 citations In Library
RiddleBench: A New Generative Reasoning Benchmark for LLMsDeepon Halder, Alan Saji, Thanmay Jayakumar et al.2025 · Conference of the European Chapter of the Association for Computational Linguistics · 2 citations In Library
LLMs for Relational Reasoning: How Far are We?Zhiming Li, Yushi Cao, Xiufeng Xu et al.2024 · 2024 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code) · 62 citations In Library
LLMs for Relational Reasoning: How Far are We?Zhiming Li, Yushi Cao, Xiufeng Xu et al.2024 In Library
War of Thoughts: Competition Stimulates Stronger Reasoning in Large Language ModelsYibin Chen, Jinyi Liu, Yan Zheng et al.2025 · Annual Meeting of the Association for Computational Linguistics · 0 citations In Library

Peer review

RejectICLR 2025 · via OpenReview
Reviewer wqH13/10 · confidence 4/52024-10-30

Summary

This paper introduces WILT as a new benchmark designed to assess LLMs in their ability to handle multi-turn reasoning tasks. Inspired by the classic Wason 2-4-6 task, WILT challenges models to deduce hidden rules by proposing test cases over several turns. Unlike typical single-turn benchmarks, WILT is structured to discourage simple memorization and requires genuine inductive reasoning by forcing models to iteratively narrow down possible hypotheses. Contributions include 1. A New Benchmark: WILT is specifically designed to test LLMs on tasks that require gathering information from previous context and adjusting responses across multiple turns. This makes it relevant for real-world applications where models engage in human-robot interactions. 2. Highlighting Multi-Turn Challenges in LLMs: The authors tested a range of current LLMs on WILT, finding that even the best models achieved only limited performance. This low performance emphasizes that many models struggle with reasoning tasks that span multiple interactions, showing a significant gap between single-turn and multi-turn capabilities. 3. Insights into Model Strengths and Weaknesses: The paper reveals interesting model-specific strengths; some models are better at generating test cases that effectively reduce the hypothesis space, while others are better in drawing conclusions from the evidence once it’s gathered.

Strengths

*Originality*: WILT shifts from traditional single-turn benchmarks to a more complex multi-turn setting by drawing inspiration from the Wason 2-4-6 task, which is an original task design. *Quality*: The benchmark aligns with research questions through clear test cases and thoughtful metrics. The analysis on model behaviors (e.g., strengths in hypothesis reduction vs. deductive reasoning) adds depth to the findings. *Clarity*: The use of illustrative examples of WILT’s structure, including failure modes (e.g., “doom loops”), aids in understanding the benchmark design. *Significance*: By identifying limitations in current LLMs' multi-turn reasoning capabilities, this paper raises an important point: strong single-turn performance does not ensure multi-turn effectiveness. This is crucial for applications requiring iterative evidence gathering and hypothesis testing, marking a step forward in aligning LLM capabilities with practical, interactive use cases.

Weaknesses

1. Limited Novelties in Benchmark Design: WILT essentially reconfigures a classic logic game, the Wason 2-4-6 task, into a format for LLMs. This design, though interesting and original, does not present sufficient novelty. Designing benchmarks based on one specific task is completely acceptable, but the selected task should at least offer unique insights, constraints, requirements that other tasks could not provide. But in this case, WILT is based on one logic puzzle and could be replicated with many other similar logic games that require inductive reasoning abilities, e.g. any tasks that require pattern abstraction could also be applicable like 20-question game, word puzzle, MasterMind etc. At least, a Motivation section should be included to justify the choice of Wason 2-4-6 task instead of others. WILT’s setup may thus be perceived as overly simplistic and lacks the depth or adaptability that would make it a significant advance over existing reasoning tests. A more comprehensive testbed with several such tasks could lead to much more convincing results. 2. Limited Insights into Model Behaviors: The paper offers insights into why models perform poorly on WILT, but does not provide practices or recommendations on potential improvements. Without deeper analysis of why specific failure modes (e.g., “doom loops”) occur or how multi-turn reasoning could be enhanced, WILT risks being seen as a "toy" benchmark, yielding performance metrics without furthering understanding. Including more potential training data augmentations or improved training techniques to counteract these limitations would add more value. 3. Inadequate Justification of Real-World Relevance: The authors justify WILT by referencing real-world multi-turn interactions, as in “mirroring real-world tasks like debugging code or reasoning over time”, yet the benchmark itself does not closely simulate real-world tasks. In actual applications, multi-turn conversations often involve at least two subjects, rich context, existence of ambiguities,varying user intentions, etc. which are not addressed by a task as simple as deducing a hidden function. WILT’s game-like setup may fail to reflect the practical challenges models encounter in real-world inductive reasoning. Incorporating richer,or contextualized scenarios, reasoning tasks could make WILT more aligned with actual LLM use cases.

Questions

1. The results suggest that models struggle with “doom loops” and “confirmation bias”, yet the analysis does not explore the underlying causes. Could the authors clarify any insights of model design issues (e.g., training data) that may lead to these failure modes? 2. “We release two test suites: a lite split, with 10 very easy tests and a canonical full split with 50 moderately difficult tests”. How were the specific rules selected? How do you define the “difficulty” of these rules? Why are the test samples so few?

Rating

3

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Authorsrebuttal2024-11-21

We thank the reviewer for their feedback. The reviewer correctly notes that other logic games could be adapted similarly. We selected WILT because test cases and guessed functions could be easily verified without relying on LLM-as-judge (e.g. 20 Questions), it does not have a potentially ill-suited visual component (e.g. Mastermind), it cannot be solved in a single turn (e.g. word puzzles), and is easy to understand despite being potentially challenging. We note that there are not many multi-turn-first benchmarks, and in our view this aspect (rather than the task) is what constitutes sufficient novelty. With respect to the origins for doom loops, we point out that many tested models are closed-source, and nearly all use non-open-source datasets even if the weights are ultimately open sourced. As a result, any analysis we could perform on why these models exhibit these behaviors would be primarily rooted in speculation, and therefore seem to us to be outside the scope of this work. We share the belief that this would be interesting, but believe analysis on preventing these behaviors is better suited for a paper on post-training for multi-turn capability, rather than a benchmark measuring the strength of multi-turn capability. With respect to the size of the benchmark, we would like to point out that the multi-turn nature of WILT makes the required number of LLM calls quite large. While 50 tests may seem small, each test requires up to 30 interactions, making the burden of running this benchmark comparable to a single-turn benchmark with 1500 examples. As the purpose of this benchmark is to easily measure multi-turn capability, adding many more tests would make the cost of running this benchmark potentially prohibitive. Regarding human baselines: we ran a pilot test on the lite split with a small set of human participants. We saw a max performance of 8/10 (matching the highest-performing LLM, Claude 3.5 Sonnet), a median performance of 7/10, and a standard deviation of 0.74. As in the original Wason Task, humans will often fail on WILT due to confirmation bias, e.g. using only 4 test cases and “confirming” their results rather than seeking falsification. We will add this in the final revision of the paper.

Area Chair PPgE2024-11-25

[Reminder] Response to Authors

Dear Reviewer, As the rebuttal period is drawing to a close, I would appreciate your response to the authors' rebuttal at your earliest convenience. Best Regards, Area Chair

Reviewer wqH12024-12-02

Response to Rebuttal

> As a result, any analysis we could perform on why these models exhibit these behaviors would be primarily rooted in speculation, and therefore seem to us to be outside the scope of this work I don't think this fall outside the scope of this work. I still believe that the task is a very simple one, thus readers would like to see you provide enough insights on why the models perform bad on the task to justify the meaningfulness of your test.

Reviewer jML85/10 · confidence 4/52024-11-02

Summary

This paper introduced a multi-turn inductive logic benchmark for LLMs. LLMs are supposed to interact with a black-box oracle by suggesting test cases. The oracle will provide the feedback on the outcome of the test cases. At the end, LLMs have to infer the hidden function based on all the test case outcomes. The paper found that LLMs still struggle with the proposed task with SoTA models achieving only 28% accuracy.

Strengths

1. The paper is clearly presented and well written. 2. The benchmark is simple and easy to use for testing LLMs. 3. The authors conducted extensive ablations on hypothesis space reduction, inversion capability, and response complexity.

Weaknesses

1. While the authors motivate the paper by claiming that LLMs struggle with multi-turn reasoning, there is no clear evidence presented to demonstrate that LLMs struggle more in the multi-turn cases than the single-turn scenarios. 2. The assumption that successful hypothesis space reduction will lead to a better chance of inferring the solution is not clearly established. In fact, Figure 2 shows quite the opposite, where all models converge to almost the same level of cumulative explorations after 10 attempts. 3. The sample size of the benchmark is too small. In addition, we should have human baselines to highlight how challenging the task is to humans.

Questions

The results from swapped test cases in Figure 3 are very hard to explain. If all models end up at about the same level at reducing hypothesis space (Figure 2), then why do the test cases matter so much at the inference time for different models?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Authorsrebuttal2024-11-21

We thank the reviewer for their feedback. With respect to struggling more in the multi-turn case, we point out that this is largely demonstrated in our paper, where strong single-turn models like o1-mini significantly outperform even the best multi-turn models when directly provided a larger suite of tests (see section 4.3). In addition, multi-turn failures such as the “doom loop” are commonly reported failure patterns in real-world applications. With respect to Figure 3, our intent with Figure 2 was to show the degree to which models stop exploring after just a few turns. We point out that in the figure there is a sizable gap between weaker multi-turn models like deepseek-chat, and stronger multi-turn models such as Claude 3.5 Sonnet. Looking at Figure 3, we note that the best test cases come from chatgpt-4o-latest, which matches the highest final point on Figure 2. In addition, models which plateau lower than the better-performing models (e.g. deepseek-chat) perform fairly well when provided other tests, since they are now being evaluated in a single-turn fashion, but other models provided their tests do poorly (since they are relying on deepseek-chat’s multi-turn capability). We believe these two figures tell a clear story, but we will attempt to communicate the findings of Figure 2 more clearly in the final paper. With respect to the size of the benchmark, we would like to point out that the multi-turn nature of WILT makes the required number of LLM calls quite large. While 50 tests may seem small, each test requires up to 30 interactions, making the burden of running this benchmark comparable to a single-turn benchmark with 1500 examples. As the purpose of this benchmark is to easily measure multi-turn capability, adding many more tests would make the cost of running this benchmark potentially prohibitive. Regarding human baselines: we ran a pilot test on the lite split with a small set of human participants. We saw a max performance of 8/10 (matching the highest-performing LLM, Claude 3.5 Sonnet), a median performance of 7/10, and a standard deviation of 0.74. As in the original Wason Task, humans will often fail on WILT due to confirmation bias, e.g. using only 4 test cases and “confirming” their results rather than seeking falsification. We will add this in the final revision of the paper.

Reviewer jML82024-11-22

I am not convinced that `there is a sizable gap` in Figure 2. Other than the two deepseek models, all the other models converge to 92-95% range. The correlation of exploration to the final performance doesn't seem to stand.

Reviewer srGe3/10 · confidence 3/52024-11-04

Summary

The paper starts with pointing out that LLMs have trouble in reasoning tasks that require gathering evidence over multiple turns and drawing logical conclusions from this evidence. The paper presents a benchmark, called WILT (Wason Inductive Logic Test), which the authors frame as a "multi-turn reasoning" benchmark, and which is supposed to be aimed at resisting memorization. In WILT, each test starts with an initial instruction. Over several turns, models must interact with the environment by suggesting test cases to narrow the possible hypotheses and ultimately infer the hidden function based on the outcomes. The experiments done in the paper suggests that LLMs struggle with this task, with the best performing model achieving only 28% accuracy.

Strengths

The paper presents an interesting task based on the task presentedd in Wason 1960, in the Quarterly Journal of Experimental Psychology. The goal of that paper was to "examine the extent to which intelligent young adults seek (i) confirming evidence alone (enumerative induction) or (ii) confirming and disconfirming evidence (eliminative induction), in order to draw conclusions in a simple conceptual task. The experiments there were designed so that use of confirming evidence alone will almost certainly lead to erroneous conclusions because (i) the correct concept is entailed by many more obvious ones, and (ii) the universe of possible instances (numbers) is infinite." In this paper, after discussing the dataset, various analysis of LLMs response to those tasks are discussed.

Weaknesses

It is not clear why the presented task is important from the point of view of an LLM's reasoning ability, as you claim? Reasoning is a catch-all phrase, so it will be helpful to elaborate on the specific kind of reasoning that you are focussing on. Does this task (and the reasoning ability that you are focusing on) have direct relationship with a task which we ask an intelligent agent to do? In particular, please provide specific examples of real-world tasks that this benchmark relates to. Also, how well do humans do with respect to the task presented in this paper? Please include a human baseline in your experiments, and/or discuss existing literature on human performance on similar tasks. Perhaps you can provide an analysis of the computational complexity for both the ideal solution and for the approach taken by LLMs. This would help contextualize the difficulty of the task. It would be helpful to give a detailed transcript of an interaction of an LLM on one of the tasks. Please provide a few representative examples showing both successful and unsuccessful interactions, along with analysis of what led to success or failure in each case.

Questions

See the questions in the weakness part. In the Caption of Figure 1, you say: "For each test, the test harness initializes a hidden rule, and participants propose up to 30 test cases for each hidden rule before making a final guess." Examples of Harness in figure 1 are given as: (x < y < z) and (x = y = z). But the harness example in tables 7, 8, 9, 10 are just triplets of numbers. Please clarify the relationship between the hidden rules (like x < y < z) and the triplets of numbers shown in the tables 7-10.

Rating

3

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Authorsrebuttal2024-11-21

We thank the reviewer for their comments. With respect to test cases vs hidden rules: the goal of WILT is to propose inputs to a black-box function, to observe the outputs, and to infer the contents of the black box. As a result, the participant proposes triplets of numbers (e.g. 2,4,6), which are evaluated against a hidden rule (e.g x<y<z), with the result returned to the participant (e.g. True). Table 7, for example, the hidden rule is “lambda x,y,z: x>0 and y>0 and z>0”. The harness column is evaluating the triplet posed by the participant, and returning the result to the participant: for example “Test Case: (1,1,1)” -> “(1, 1, 1): True” Regarding the real-world importance of this task: the purpose of WILT is to measure the capability of language models on simple tasks in explicitly multi-turn settings. To make an analogy, most benchmarks act like “chess puzzles”, where the model can make many logical steps in one turn in order to arrive at a goal state. WILT by comparison is like a “chess game”, where arriving at the correct answer in one step is not possible, and the model needs to repeatedly pose questions and gather evidence. This has implications for many LLM use cases which are more explicitly multi-turn, for example: debugging code, diagnostic reasoning, tutoring and education, multi-turn roleplay conversations, and so on. Our view is that multi-turn capability is understudied, despite its practical importance, hence this benchmark. We point out that Tables 7, 8, and 9 represent the requested transcript, only excluding the LLM’s reasoning processes and the system prompt in Appendix A.1.2. Table 7 represents a successful case, where the model repeated several tests but ultimately still arrived at the correct conclusion. Table 8 represents a case where the model gathers appropriate evidence to solve the test, but adds unnatural bounds which cause it to fail. Table 9 represents a model which refuses to use additional turns to gather more evidence, which is an explicit multi-turn failure. Regarding human baselines: we ran a pilot test on the lite split with a small set of human participants. We saw a max performance of 8/10 (matching the highest-performing LLM, Claude 3.5 Sonnet), a median performance of 7/10, and a standard deviation of 0.74. As in the original Wason task, humans will often fail on WILT due to confirmation bias, e.g. using only 4 test cases and “confirming” their results rather than seeking falsification. We will add this in the final revision of the paper. Regarding a computational complexity analysis of the task, we candidly admit that the hypothesis space metric is an approximation of how quickly LLMs can clear out this space. The purpose of this metric is to provide a relative comparison, demonstrating that some LLMs completely abandon the exercise of reducing bad hypotheses.

Area Chair PPgE2024-11-25

[Reminder] Response to Authors

Dear Reviewer, As the rebuttal period is drawing to a close, I would appreciate your response to the authors' rebuttal at your earliest convenience. Best Regards, Area Chair

Reviewer srGe2024-11-25

Still not convinced that this is a suitable task for LLMs.

I humbly disagree with the "simple task" part of your statement "the purpose of WILT is to measure the capability of language models on simple tasks in explicitly multi-turn settings. To show that this is a simple task, giving an algorithm to solve this would be helpful.

Reviewer pqpf6/10 · confidence 4/52024-11-04

Summary

The paper introduces WILT (Wason Inductive Logic Test), a new reasoning benchmark for LLMs to evaluate reasoning over multiple turns. Inspired by the Wason 2-4-6 task, LLMs need to identify a function of three variables by proposing test cases and observing outcomes. The authors evaluate all the main state-of-the-art LLMs on WILT and find that they struggle with the task, achieving a maximum accuracy of only 28%. The authors also perform additional analyses, looking into hypothesis space reduction, response complexity, and test case novelty.

Strengths

* Novel Benchmark: WILT addresses an important gap in LLM reasoning evaluation by focusing on multi-turn reasoning. This capability is important for real-world applications like chatbots. * Synthetic examples: The design effectively prevents models from relying on memorized responses from training data. * Broad Evaluation and interesting analysis: A wide range of LLMs is evaluated. Additionally, the authors provide a detailed analysis of their performance using multiple metrics, including accuracy, hypothesis space reduction, response complexity, and test case novelty. The analysis reveals interesting insights into the models' behavior. * Reproducibility: The authors provide clear descriptions of the benchmark, experimental setup, and evaluation metrics, facilitating reproducibility (even if the code is not released, which they promise to do).

Weaknesses

* Limited scope of reasoning: The tests in the benchmark are limited to simple arithmetic and logical operations. Testing multi-turn reasoning on a wider range of reasoning tasks would be interesting. * Lack of human baseline: It would be interesting to know how good humans are at this task. * 'Approximately correct' metric: As far as I can tell, there is no clear definition of this metric.

Questions

1. Can you provide more details on how the functions are generated? 2. An analysis by function type would be interesting. 3. Do you have an idea of how hard this task would be for humans? Did the authors try themselves on a subset? 4. Can you provide a precise definition for the "approximately correct" metric?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Authorsrebuttal2024-11-21

We thank the reviewer for their thoughtful comments. With respect to the scope of reasoning: we agree that the tests in the benchmark are very simple. This is by design - our intent was to create a maximally difficult multi-turn task while keeping the single-turn component as deliberately simple as possible. Our task domain of arithmetic and logical operations ensures that guesses can be clearly verified. Other benchmarks such as BIG-BENCH 20Q provide tests in the semantic domain and could be used in conjunction with WILT to support any trends in performance. However, such semantic tasks often result in the LLM asking questions with debatable answers, thus reducing the standardization of the tests. Regarding the “approximately correct” metric: a response is considered “approximately correct” when it captures the core logical relationship but fails on edge cases - for example, proposing x < y < z when the true rule is x ≤ y ≤ z. These are intended to provide resolution whether models can learn important behaviors about the underlying rule, but fail to appropriately test their way to the correct rule. Regarding human baselines: we ran a pilot test on the lite split with a small set of human participants. We saw a max performance of 8/10 (matching the highest-performing LLM, Claude 3.5 Sonnet), a median performance of 7/10, and a standard deviation of 0.74. As in the original Wason task, humans will often fail on WILT due to confirmation bias, e.g. using only 4 test cases and “confirming” their results rather than seeking falsification. We will add this in the final revision of the paper. Again, we thank the reviewer for their time and attention.

Reviewer pqpf2024-11-24

Thank you for the response, and for testing the task on a small set of humans. If you haven't made that change already, please provide a complete definition of "approximately correct" in the paper.

Area Chair PPgEmeta-review2024-12-20

Meta-review

The paper introduces WILT (Wason Inductive Logic Test), a new benchmark designed to evaluate the multi-turn reasoning capabilities of Large Language Models (LLMs). Inspired by the classic Wason 2-4-6 task, WILT requires LLMs to infer a hidden logical or arithmetic function of three variables by proposing test cases and observing the outputs. The authors evaluated various state-of-the-art LLMs on WILT and found that they struggle significantly with this task, achieving a maximum accuracy of only 28%. The paper further investigates model behavior by analyzing hypothesis space reduction, response complexity, and the novelty of proposed test cases. The core finding is that current LLMs, despite their strength in single-turn tasks, exhibit limitations in effectively gathering evidence and reasoning across multiple interactions to solve this inductive logic problem. The main problem of this paper stemmed primarily from the reviewers' doubts about the value and impact of WILT as a benchmark. They questioned whether success on this specific task truly reflects meaningful progress in LLM reasoning, citing the limited scope and unclear connection to practical applications. The perceived lack of significant novelty and the absence of a human comparison point further weakened the justification for the benchmark, leading to the conclusion that it did not sufficiently advance the field. While the benchmark presents some interesting ideas, its potential for realistic impact appears limited. Therefore, based on this assessment of its practical applicability and broader contribution, I lean towards rejection of this paper.

Additional comments on reviewer discussion

All reviewers respond to the authors' rebuttal and three of them agree on the rejection of this paper. Therefore, based on this significant agreement, the decision is to reject the paper.

© 2026 NYSGPT2525 LLC