Weak-eval-Strong: Evaluating and Eliciting Lateral Thinking of LLMs with Situation Puzzles

Summary

The paper constructs the benchmark SPLAT to evaluate and elicit lateral thinking abilities in LLMs. It includes questions similar to brain teasers, which describe scenarios and ask non-obvious questions for guessing. Example: __Question/Story__: A hunter aimed his gun carefully and fired. Seconds later, he realized his mistake. Minutes later, he was dead. __Reference Answer__: He hunted in snow-capped mountains. The shot provoked an avalanche, which covered the man. He died of strangulation. They also propose a multi-turn player-judge evaluation framework that reduces reliance on stronger evaluation models.

Strengths

1. The SPLAT dataset is different from previous datasets, with graded difficulty levels and harder questions. 2. the multi-turn player-judge framework are claimed to be able to reduces reliance on stronger evaluation models. (I acutally feel a bit confused for this part, see qustions) 3. The authors conduct experiments to validate their approach, demonstration that SPLAT can improve LLM performance on other lateral thinking tasks suggests broader applicability.

Weaknesses

1. "The dataset and framework are designed not only to evaluate but also to actively elicit lateral thinking in LLMs. Experiments show that using data and reasoning processes from our framework leads to improved performance of LLMs, even when applied to other lateral thinking benchmarks." This part really needs ablation studies. It's unclear which parts are effective. For example, are the reasoning processes, the entire dataset, or similar questions in previous data responsible for the improvement? Did this data improve the model's ability to think about these types of questions, or is the improvement possibly due to some tricks? Moreover, why was testing only done on RiddleSense when many other benchmarks were reported earlier? Is this cherry-picking? 2. A player-judge evaluation framework based on WizardLM-2 is proposed as an alternative to human judges to evaluate the answers, but it shows much worse agreement on hard questions compared to inter-human agreements. However, harder questions are actually proposed as a contribution of this work. As the answers for such puzzles can be "off by a hair but miss by a mile," it might require human evaluation for the hard puzzles. It is also a bit confusing why the human agreement on Medium questions is much lower than on hard and easy ones.

Questions

1. Similar datasets have existed before, as mentioned in the paper, such as RiddleSense and Oogiri. Here's an example from RiddleSense: My life can be measured in hours. I serve by being devoured. Thin, I am quick; Fat, I am slow. Wind is my foe. What am I? (A) paper (B) candle (C) lamp (D) clock (E) worm In comparison, SPLAT has longer questions and answers. Also, the reference answers for SPLAT are open-ended, while previous ones are all multiple-choice, which is claimed as a contribution. However, this also makes the evaluation more challenging. What makes you think open-ended is a much better option that makes it a contribution? 2. Author claim that their methods reduces reliance on stronger evaluation models, but it seems they are still using WizardLM-2 as a component (Judge Model) of their framework?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The size for this benckmark is small compared with previous similar datasets that have existed before, which include many more puzzles compared to the proposed SPLAT.

Summary

This paper focuses on lateral thinking, which is about creativity and viewing problems from multiple angles. To sovle the challenge that the complexity of assessing creative thought processes and the scarcity of relevant data, this paper introduces SPLAT, a benchmark leveraging Situation Puzzles to evaluate and elicit LAteral Thinking of LLMs. By employing a new multi-turn player-judge framework instead of the traditional model-based evaluation, the proposed method lessens dependence on more robust evaluation models, enabling the assessment of state-of-the-art LLMs. Moreover, through applying data and reasoning processes from our benchmark to another lateral thinking-related benchmark, like RiddleSense, leads to performance enhancements.

Strengths

1. This paper focuses on the lateral thinking problem, which is important to creativity and viewing problems from multiple angles. 2. The proposed dataset is open-ended and the evaluation method is good. 3. The paper is well-written.

Weaknesses

1. The size of the dataset is samll. 2. The reference answer is limited to 1, while the questions are one-to-many questions. 3. The in-depth analysis is not enough

Questions

1. Why not provide more than one answer for each question (I do not think it is challenging, there are multiple answers to a question on the internet such as Quora)? Since one such question could have multiple answers, more reference answer might also be helpful for evaluation. 2. A overall evaluation metric is needed, some model might have a high acc with a high average round, you should define a overall metric to combine these two metrics for a clear comparison of existing LLMs. 3. It is better to adopt more benchmarks for validating the effectiveness of the proposed dataset (more than just RiddleSense).

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

None

Summary

This work focuses on creating a benchmark, as well as a modeling framework for evaluating lateral thinking of LLMs. The benchmark, called SPLAT, consists of 975 situation puzzles. The framework consists of a “judge” and a “player” (the LLM to be evaluated). The judge poses an open ended puzzle that is further clarified by the player till a final answer is reached. The judge evaluates the final answer. The authors posit that this framework eases the challenge faced with typical auto-evaluation models since the judge does not need to be better than the LLM that is being evaluated. The authors evaluate the quality of the judge (via human agreement), and finally evaluate LLMs using a promising judge.

Strengths

The motivation for having a lateral thinking benchmark is strong. The work around automatic evaluation using a judge v/s player framework is well thought out. I think such a benchmark as well as framework/protocol would be beneficial to the community.

Weaknesses

Overall, the idea and motivation behind the work is solid. I also think that the benchmark is good quality. However, my concern is that this work is half-baked. The experiments aren’t conclusive. More work should go into making the experiments thorough and useful to the scientific community. * Although the motivation and overall idea is strong, I am concerned about the experimental setup. The agreement rates for WizardLM-2 in Table 2 and 3 don’t appear to be high enough (are around ~80%) and 3 individuals seems to be fairly less. How does this fare with human agreement rates in literature? Also, is there any reason we haven’t used the strongest model (GPT-4 from the paper) as the judge? * I also am concerned about Table 4. The authors should consider using strong base models for evaluation (that are closer to GPT-4 on public benchmarks) * It would be good to introduce modeling methods to improve upon the metrics that are obtained for models on SPLAT. This will give more insight into how this benchmark is valuable to the community. * nit: I see mentions of “using data and reasoning processes from our framework” consistently throughout the paper. This should be framed better and be more precise.

Questions

See weaknesses mentioned above.

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

3

Limitations

NA

Summary

This paper introduces SPLAT, a novel benchmark for evaluating and eliciting lateral thinking abilities in Large Language Models (LLMs) using situation puzzles. The key contributions include: A new dataset of 975 graded situation puzzles across three difficulty levels. A multi-turn player-judge evaluation framework that reduces reliance on stronger evaluation models. Demonstration that the benchmark can both evaluate and elicit lateral thinking in LLMs. Experimental results showing that a robust evaluation model (WizardLM-2) closely matches human judgments. Evidence that applying SPLAT's data and reasoning processes to other lateral thinking benchmarks leads to performance improvements in LLMs.

Strengths

1. The paper presents a novel approach to assessing lateral thinking in LLMs, an area that has been underexplored compared to vertical thinking. The use of situation puzzles and the multi-turn player-judge framework are creative solutions to the challenges of evaluating open-ended, creative thought processes. 2. The methodology appears rigorous, with careful data collection, annotation, and difficulty categorization. The multi-turn player-judge framework is well-designed to overcome limitations of traditional model-based evaluations. The experimental results, including comparisons with human judgments and applications to other benchmarks, provide strong evidence for the effectiveness of the approach. 3. The paper is well-structured and clearly written. The task definition, data construction process, and evaluation framework are explained in detail. Figures and tables effectively illustrate key concepts and results. 4. This work addresses an important gap in the evaluation of LLMs by focusing on lateral thinking, which is crucial for creative problem-solving and innovation. The benchmark has potential applications beyond evaluation, as demonstrated by its ability to improve LLM performance on other lateral thinking tasks. This could lead to advancements in developing more creative and flexible AI systems.

Weaknesses

1. While the data collection process is described, there's minimal discussion of potential biases in the dataset, such as cultural specificity of the puzzles or biases introduced during the human annotation process. 2. The paper would benefit from ablation studies to isolate the impact of different components of the SPLAT framework, such as the difficulty categorization or the multi-turn questioning process. 3. The paper lacks a discussion of potential ethical implications or misuse of improved lateral thinking capabilities in LLMs.

Questions

1. How does the performance of LLMs on SPLAT correlate with their performance on other, more traditional NLP tasks? Is there evidence that lateral thinking ability is distinct from other language model capabilities? 2. Have you considered ways to automatically generate new situation puzzles to expand the dataset and improve scalability? 3. How sensitive is the performance of LLMs on SPLAT to the specific prompts or instructions given? Could you elaborate on the prompting strategy used?

Rating

8

Confidence

5

Soundness

4

Presentation

3

Contribution

4

Limitations

None.

Replying to Rebuttal of Authors

Thanks for your detailed response. It has addressed some of my concerns, I will raise the score to reflect this. However, I am still concerned that the number of answers to each question is limited to 1 since your paper is about "Lateral Thinking". Although "How to build a situation puzzle?" is not proper to obtain multiple answers, there still are many questions that have answers from multiple angles, such as "how to prove 1+1=2".