Cryptic crossword clues are challenging language tasks for which new test sets are released daily by major newspapers on a global basis. Each cryptic clue contains both the definition of the answer to be placed in the crossword grid (in common with regular crosswords), and 'wordplay' that proves that the answer is correct (i.e. a human solver can be confident that an answer is correct without needing crossing words as confirmation). This work describes an LLM-based reasoning system built from open-licensed components that solves cryptic clues by (i) hypothesising answers; (ii) proposing wordplay explanations; and (iii) using a verifier system that operates on codified reasoning steps. Overall, this system establishes a new state-of-the-art performance on the challenging Cryptonite dataset of clues from The Times and The Telegraph newspapers in the UK. Because each proved solution is expressed in Python, interpretable wordplay reasoning for proven answers is available for inspection.
Paper
Similar papers
Peer review
Summary
The paper introduces a novel reasoning approach to generate answers for cryptic crossword clues. It is a multistage approach which does the following: 1. First, a fine-tuned LLM suggests possible answers for the entire clue. 2. Second, for each candidate answer, another LLM generates possible definition/wordplay suggestions. 3. Third, for each suggestion, another few-shot LLM generates python code which wraps their domain-specific language to prove the wordplay. If there's an error in the verifier, it is fed back to the LML in a loop up to K times to improve/correct the proof. If there is no error, the answer is used as the final prediction. This system improves overall performance on the Cryptonite dataset from 15.7% to 32.5%, and the authors give some analysis on the benefits and shortcomings of their approach.
Strengths
* The authors introduce a novel approach to a uncommonly studied task, which has interest to the NLP/reasoning communities for its difficulty and unique reasoning requirements, and to the cryptic crossword hobbyist community. They provide motivation for why this task is relevant and should be worked on. * Their approach is effective, more than doubling the previous SOTA performance on the same task. Additionally, it offers several "obvious" avenues for improvement as it breaks the task down into more easily tunable parts. * Finally, the paper is very clearly written and communicates their relatively unknown task and novel method effectively.
Weaknesses
* A full ablation study would make clear the shortcomings of each part of the pipeline. While the authors demonstrate the effectiveness of just using the top candidate answer, it would be helpful to know the benefit of adding the definition/wordplay model (combined with e.g., a simple LLM-based reranker) compared with the full pipeline.
Questions
- When you say `The Wordplay dataset follows the train, validation, and test split defined by Cryptonite`, I assume you mean there is no contamination between the two, i.e., no clue in the train dataset for Wordplay will appear in the Cryptonite val/test splits. Is this correct or did you have mean something different? - The performance of your pipeline is of course capped by the recall of the candidate answer suggestion model, which seems to be about $40$% at $N=20$. This means that the rest of your pipeline can correctly choose about $32.5 / 40 \approx 81$% of the results, which seems promising. Have you considered just making N as large as possible? It might be an easy (albeit inefficient) way to improve overall performance by a bit, assuming the rest of the pipeline is robust to many wrong candidates. - Also, I'd be curious to know how much the 'verification' system helps. While your system is certainly effective, I wonder how much the verification system improves over a "dumb" reranker (e.g., scoring the probability of each (answer, definition/wordplay) pair using the LLM and choosing the best).
Rating
8
Confidence
4
Soundness
3
Presentation
4
Contribution
3
## Re: Weaknesses ### Pipeline Ablation We agree that doing an ablation using a trained re-ranker for each of the definition / wordplay steps would make an excellent addition to the paper (please see below). ## Re: Questions ### Cryptonite vs Wordplay splits Yes : The splits of these two datasets has been made deliberately consistent (i.e., no clue in the train dataset for Wordplay will appear in the Cryptonite val/test splits). Moreover, the code for generating the wordplay dataset intentionally does not output wordplay for the test split - to avoid easily digestible test set data appearing in the wild as a result of the construction of the wordplay dataset. ### Number of candidate answers The choice of N=20 was, frankly, mostly one of computation/time resources. We would be happy to add a graph of the frequency of the correct answer being in the top-N for different N (the data up to 20 already exists). However, our observation has been that (supposing the most-frequent answer is not the correct one), there is a balancing act to be performed : Choosing a non-most-frequent answer (for large N) becomes very risky, since the 'tail' of the answers (across the English language) is rather long, and the proofs are not iron-clad (since the proving DSL includes functions such as 'is_synonym()' - which is rarely a solid Yes/No decision, see the "EXAMPLE" example on page 1 : Here, 'case' is a true, but weak, synonym for "EXAMPLE" ). Thus, trying for large N (indeed, one could imagine just throwing all '7 letter words' at the proving system) would conceivably cause more harm than good. ### Pipeline Ablation : Use of a 'dumb' reranker Many thanks for this suggestion - this would clearly be a good addition the paper. Anecdotally, to some extent we tested the first of these steps (choosing a candidate answer) by having the model make multiple suggestions until it chose to halt (trained to halt at the first correct suggestion). This was somewhat effective (though the experiment was directed at whether self-correction could be trained for - as part of another line of investigation). The second step (whether proposed wordplay 'makes sense', independent of the formalisation/prover) has also been informally tested - given the observation (illustrated in Section 4.2 of the paper) that wordplay for incorrect answers is rather clearly absurd. There is an additional interesting question, though, of whether 'provable' vs 'non-provable' wordplays for a correct candidate answer can be distinguished without the help of the Formaliser+Python elements. This, perhaps, may be difficult to train for (given the limited number of successful proofs overall). We will push to get the first two ablations done by the end of the review period - many thanks for the suggestion.
Summary
This paper identifies the gap that cryptic crosswords, a well-known language-oriented reasoning puzzle, have received little attention from the community. To this end, the authors propose a system that combines LLMs and a Python interpreter to solve cryptic clues. In the paper, they elaborate on the system architecture and run experiments using Gemma and Gemini model family to show better performance than the baselines on Cryptonite benchmark.
Strengths
Identifying the gap that cryptic crosswords have received little attention from the community, especially post LLM era, the authors propose a framework that combines LLMs and a Python interpreter to solve cryptic clues, and show better performance than baselines on Cryptonite benchmark.
Weaknesses
1. This paper may lack enough strong baselines and the comparisons with these baselines may not be entirely fair. There are 3 baselines including rule-based, a fine-tuned T5 large, and Gemini Pro 1.0 zero-shot. First, The T5 large model only has 770M parameters while Gemma 9b (which is used in this paper) has 9B parameters. Comparing Gemma 9B-FT with T5, we can see that changing the model with the same training data improved the accuracy from 7.6% to 15.9%. Second, Gemini 1.5 Flash is used in the proposed system while the baseline is zero-shot Gemini Pro 1.0 which shows inferior performance across general benchmarks. Third, while the papers mentions limited computation resource, rows in table 1 are reported based on different number of examples, which make them not comparable. 2. It may not be clearly stated in the paper the reason of following the proposed order: i.e. generating answer candidates first, then generating possible definitions and wordplays, and then doing verification. Reasoning tasks typically generate reasoning steps followed by the final answer and verification, e.g. ReAct format. In this case, possible definitions and wordplays can be followed by the final answer candidates and external verifiers. This is also consistent with line 44 regarding the reasoning steps. Have you tried different variations and concluded that the proposed order is the best?
Questions
1. Regarding weakness 2, have you tried different variations and concluded that the proposed order is the best? 2. Since the current system combines multiple open source fine-tuned models (Gemma) and a closed source model (Gemini) for different stages, I wonder if you have tried a single model performing all different tasks, either prompting or fine-tuning, which makes the system simpler? 3. The last paragraph in section 2.2 mentions that an apple-to-apple comparison with Rozner et al. (2021) is hard because of different split setting. Could you elaborate more on the challenge of comparing using the same data? Is it because the test split in one approach is used for training of another approach, or their model is not publicly available? Minor points => Typo(s): 1. Line 155, T5 large is 770M 2. Line 200, the first sentence in section 3.1 has two “from” 3. Line 344, that -> than
Rating
3
Confidence
4
Soundness
1
Presentation
2
Contribution
1
## Re: Weaknesses ### Baselines Many thanks for catching the size of T5-large (11B->770M) - this makes the parameter counts and scores make more sense, since (despite having been trained in 'earlier times') the T5 series are very effective models. Other papers (such as the arXiv paper of Saha et al, 2024) have details about the 5/10-shot scores achieved by Mistral 7B, Llama3 8B, Mixtral, Llama2 70B and Llama3 70B (these all score below the rule-based Deits figures reported). Note that the 770M T5 model was fine-tuned on the Cryptonite training set, rather than being used to predict answers in 5-shot (or 10-shot) way. In terms of baselines, the Gemini-Pro 1.0 result was used to illustrate that (at the time) commercial models had difficulty with cryptic crosswords as a whole. Clearly, the result for GPT-4-T (23.5% on Cyptonite the test set) reported in the arXiv paper of Saha et al (2024) shows that some commercial models can achieve excellent results (though it is quite possible that they have been specifically trained on Cryponite itself) - so we plan to include it in the same bracket as the Gemini-Pro 1.0 result. Our goal here, though, is not to run a competition between commercial models : Their training data is opaque, and knowing which brand currently performs best is of limited/transitory value. The use of Gemini-Flash in this paper is in the (limited) role of a Formaliser, and the last line of Table 1 shows that it can be replaced by Gemma2-9B-it (without fine-tuning) and still achieve SoTA results. **EDIT** : We have run the additional baselines as requested - please see the "Up-to-Date Baselines" comment above. Finally, while the results in Table 1 have different numbers of samples, the accuracy rates are comparable, standard deviation measures can be added to the table to make the differences clearer. ### Order of Operations The order chosen (where answer candidates are used to generate potential reasoning chains, which are then verified) does indeed differ from (say) ReAct. This approach was used based upon watching human solvers - who report (/ observe on YouTube) going through the following steps: (a) attempt to parse the clue in a number of ways, trying to isolate the definition from the wordplay; (b) seeing which parts of the wordplay they are most confident about; (c) 'having a hunch' of the final answer; and (d) gaining a full understanding of how a clue's wordplay works (i.e. can every element be explained) as proof of the overall process. Around L44, the description of the "EXAMPLE" process has been laid out in a way that is more of a logical explanation. A more typical thought process for the clue "Cut up over politician on the French case (7)" might first recognise that "politician"="MP" and "the French"="LE" - and "on" and "over" are very positional words - leading to checking whether there's a 7-letter word for "Cut" or "case" that includes "MPLE". The final verification would see that "Cut"="AXE" and "up" is a reversal indicator (for a down clue). "EXA-MP-LE" : QED. Observing the behaviour of (say) GPT-4 (using CoT) (or more recently o1), suggests that even very capable models tend to fixate early on during the reasoning process, and are only rarely able of completely re-hypothesising. We also noticed LLMs being caught up with the literal ('surface') meaning of the clue, which is often deliberately misleading. The combination of these elements caused us to re-consider how to approach this kind of problem. Giving our process candidate answers up-front (so that they try to fit the reasoning to the answer, with varying degrees of success) bakes the re-hypothesisation in. One weakness, however, is the looseness of the language used in these puzzles - "case" and "EXAMPLE" are only easily recognisable as synonyms in a specific sense. Clearly, the idea of iterative proof refinement is very attractive (along with RL, etc), and we hope to use cryptic puzzles as a test-bed for this in future work. But, similarly to IMO problems (rather than more logically approachable mathematics reasoning, say, GSM8K or MATH) we anticipate that the 'Aha!' moment is going to be a sticking-point. It is with interest that we saw Alpha-Geometry adopt the approach of suggesting interventions, and then rolling out what-if proofs from there.
## Re: Questions ### Weakness 2 : Order of operations Discussed above. Practically speaking, the paper's order of operations constituted the bulk of the research performed (other orders were considered, but eliminated prior to major investment of effort due to anticipated short-comings). One other element being considered was the quantities of data/compute available: (a) 400k+ Cryptonite clue->answer pairs; (b) <10k clue-> wordplay pairs; (c) infinite patience for the Python prover; and (d) limited time/resources for running many long-context experiments through an LLM (despite Gemini-Flash being remarkably convenient and low $/token) ### Closed vs Open models In fact, although the Formaliser model on which most experiments/trials were conducted was the Gemini-Flash model, Table 1 (last line) shows the result for Gemma2-9B-it (without fine-tuning) on the same task. The 'headline result' decreases from 32.5% to 29.0% - which would still be SoTA, even with the updated GPT-4-T numbers. Thus the Gemma2-9B base (either fine-tuned on Cyptonite, Wordplay or its '-it' variant) are quite capable of excellent performance. As identified above (and where we may have appeared over-defensive about baselines), this does raise the question of how best to include commercial/proprietary results. ### Comparisons vs Rozner et al (2021) **EDIT** : This comment has been expanded upon below - since it raises an additional issue about commercial vendor training that deserves more attention ## Text Fixes Typo fixes (and the size of T5-large) have been incorporated : Many thanks.
### Comparisons vs Rozner et al (2021) At the start of our research program, the Cryptonite dataset of Erfat et al (2021) was chosen as being the focus, over the approximately contemporaneous dataset from Rozner et al (2021) (denoted Rozner below), for the following reasons: * Cryptonite was larger (523k clues, compared to 142k in Rozner) * Cryptonite consists of clues from The Times and The Telegraph (whereas Rozner is the UK's Guardian). While these are all fine newspapers, it is clear that in the cryptic crossword community (found online via websites for wordplay discussions, or YouTube channels) that The Times is considered the Gold Standard of cryptic crosswords. + Indeed one of the Guardian's own cryptic blog posts (https://www.theguardian.com/crosswords/crossword-blog/2024/nov/04/cryptic-crossword-ai-conquer-human-solvers-artificial-intelligence-software) directly states: "The Times hosts an annual crossword-solving competition and it remains, until such time as the Guardian has its own version, the gold standard." + In the author's view, The Times deserves its role as Gold Standard due to (a) adhering to / upholding the Ximinian standard for what is allowed in clues; (b) doing so for decades; (c) maintaining high consistency of clue difficulty within puzzles (where solvers frequently complain that the Guardian clues can often be rather haphazard) * The Cryptonite dataset was made available for direct download - even though the licensing is (politely) 'fuzzy', it remains a useable research dataset (and seems unlikely to be challenged by The Times, since it is not possible to reconstruct their full puzzles from the clues given as individual line-items, due to deduplication, for example) + The Rozner dataset required researchers to 'scrape their own data', likely because while the data was being retrieved from a public website, the data itself could reasonably be assumed to be copyrighted. This slight inconvenience had a useful impact (please see below) * Unlike the Cryptonite dataset, the Rozner dataset does not include Across/Down markers for the clues - which makes some of the clues difficult to resolve (for instance "EXAMPLE" on the paper's first page can only be read correctly if one sees that it is a Down clue - which converts 'up' into a reversal indicator) * The Cryptonite dataset splits were set in stone. Rozner, though, had a series of splits (random, disjoint, and 'init'): + The 'random' split was clearly shown to be a poor way of separating train/test due to close overlaps + The 'disjoint' split is similar in spirit to the Cryptonite methodology + The 'init' split had the additional twist that common prefixes would only be found in their own splits. This had a catchy intuition, although it's not clear from a cryptic cluing perspective whether this has much genuine basis. While there are some prefixes that are common (eg: 'EX-' is easily clued by refering to divorce, etc), the impact seems overall marginal (particularly given the accuracy rates reported) Our paper describes a system trained on Cryptonite clue/answer training data, and also (as a component) the Wordplay dataset (which abides by the Cryptonite splits too). If it is considered essential, we *could* test our existing (Cryptonite trained) system on the Rozner 'Init' test set. However, while Saha (2024) could have the flexibility to run tests on either dataset (since no training was performed), our own 'Init' test would be clearly mis-aligned vis-a-vis the data split. But there is also a structural reason against re-training the paper's system on the Rozner 'Init' split for (specifically) Wordplay. The Wordplay dataset generation process was guided by the principle of maintaining the Cryptonite splits, it would be a disaster if Rosner Init *Wordplay* splits were to be made public. The reason: The Cryptonite test set (very likely) has a large intersection with the Rozner Init training set. As seems evident from the baseline improvements shown above, OpenAI likely trains on the Cryptonite training set (they are welcome to do so). HOWEVER, since now Saha (2024) appears to be releasing the 'Init' training set under an MIT license, OpenAI would be quite within their rights to also train on that. Thus, commercial systems (against which reviewers are forcing academic papers to benchmark) will have been trained on the test sets (without commercial vendors explicitly 'cheating' - they will just be training on all the available training data). In our judgement the *reasoning paths* that are being tested here by the cryptic crossword task are a prize cultural asset, generated over decades of human effort, and this should not be squandered. Hopefully, this explains the authors' reluctance to dataset-hop : We don't want to make it common to gather and distribute cross-contaminating Wordplay datasets.
In our 3 sets of comments added directly above, and also in the 3 comment sections added for all reviewers (including the updated version of the paper that we uploaded last week), we addressed the following weaknesses that you identified, and questions that you had : * We have updated the baselines used in the paper beyond those obtained in Saha et al, (2024). The new GPT4 5-shot results (that we show in updated Table 1) are +4.1% higher than the results previously given in Saha et al's arXiv paper. + Even so, our results with the Gemini-Flash formaliser, are still well ahead of that baseline + When the whole pipeline consists of open-licensed models (with a non-finetuned Gemma2-9it acting as formaliser) we still beat the updated GPT4 5-shot results (though by a slimmer margin) + We have included standard deviation figures for the results in Table 1 - and would be happy to discuss further statistical measures that show that both results are measurably better than the baselines * We discuss the rationale for the ordering of the pipeline in the new version of the paper (at the beginning of the Methods section) + This new section of the paper is a summary of the points made in our direct answers above * The paper now includes more emphasis on the ability of our open-licensed pipeline to beat the GPT4o 5-shot baseline * With respect to measuring results against the 'Init' split of the Rozner (2021) dataset, the paper goes into more detail about why Cryptonite has been the focus of the work, and also mentions (in the Appendix) some potential hazards of having multiple datasets with cross-mismatched splits. + We particularly do not want to risk the key Wordplay dataset losing its train/val/test clarity by creating new versions that are split in such a way as to contaminate commercial model training against the Cryptonite test split (this issue is described more fully in the Appendix) Is there anything that you need from us to answer any remaining questions that you have? We believe that our new version of the paper substantially addresses the issues that you identified.
Summary
The authors propose a method that achieves state of the art performance on cryptic crosswords. The method uses LLMs to solve the subtasks using various strategies. First, Gemma2-9B is fine-tuned using low-rank adapters on answer-candidate generation, which returns a list of answers which match the 'pattern' (the length of the answer). Second, Gemma2-9B is again fine-tuned using low-rank adapters, but this time for 'wordplay suggestion', which takes each answer candidate and generates multiple definition/wordplay pairs, which are meant to explain why the answer is correct. Third, prompting is used in combination with Gemini-Flash to formalise the wordplay into Python 'proofs'. The Python proofs are run and any errors are reported back to Gemini-Flash, which is given a chance to correct its proof (until the answer is valid, or the maximum of 2 rewrites is reached). In the experiments, the formalisation process is shown to be helpful in the general, although for the 'Quick' subset, using the most frequent answer candidate proves too strong a baseline.
Strengths
- State of the art result on complex task. - The reported results show that LLMs are very helpful as components in a system to solve complex tasks, while also not sufficient to solve it completely on their own.
Weaknesses
- I think more space could be devoted to explaining cryptic crosswords. As someone previously unfamiliar with them, it took quite long for me to understand what exactly was happening in the example given in the introduction. Maybe a table or diagram that explains exactly how each part in the wordplay is related to the parts in the clue? - A solid motivation for why this work is important is missing. What is the primary reason to be interested in LLM performance on this task? And relatedly, what should be the general take-away after reading it?
Questions
How were the in-context learning prompts constructed, and how many versions did you try? What is your sense of the sensitivity of the final accuracy w.r.t. the prompt, how important is it?
Rating
5
Confidence
3
Soundness
3
Presentation
3
Contribution
1
## Re: Weaknesses ### Explanations of Cryptic Clue mechanisms We would certainly be happy to add diagrams for each of the clues used in the body of the paper ("EXAMPLE", "DELVE", "HERON", etc). And, of course, the current more detailed exposition in Appendix A should also be referenced. ### Motivation There are several motivations to study how model performance can be improved on the cryptic crossword solving task: (a) The task is something that thousands of people find a satisfying intellectual challenge on a daily basis. Solving these puzzles requires understanding multi-layered language constructs, blending logic, wordplay, and contextual nuance. This provides a unique challenge for evaluating and improving LLMs’ capabilities in NLU and reasoning. (b) There is ample data (decades of solved puzzles, each containing over 20 clues, from multiple major newspapers), including new (original) test data being created daily : This contrasts with (for instance) IMO/AIME problems, where there is a much lower number of novel problems available. (c) The raw performance figures for LLMs on this task are perhaps less interesting than the techniques required to perform the reasoning itself. The method in this paper, though this aspect wasn't emphasised, also explicitly reveals the reasoning (i.e. validated wordplay) required to solve each problem. There is typically one 'true' reasoning path that 'works', although it might be expressed in slightly different ways by different solvers. This work is important because it is the first to apply a hypothesis/reasoning/verification system to a task with this degree of flexibility. General take-away: Cryptic crosswords have been shown to be a fertile test-bed for the next generation of reasoning systems (which may include an LLM as a component, but operate in a more iterative manner). ## Re: Questions ### Construction of ICL prompts / Sensitivity The LoRA-fine-tuned Gemma models had simple instructions for the tasks (clue->candidate answer) and (clue+answer->wordplay). These were not iterated on specifically, because (following the SFT process on actual data) the Gemma models just use the prompt text as a signpost to go into 'remember the specific task' mode (in the author's mental model). Some iteration was performed for the LLM (Gemini-Flash) prompts, since it was thought that the transformation from Wordplay to Python 'assert' statements would require much more background knowledge. However, the 'prompt engineering' was more to do with the structural outline than specific word-smithing: The overall prompt starts with a (terse) introduction to the specialised 'defined terms' for cryptic puzzles; then 20 examples of wordplay (which are cheap/short); finishing with a limited number of Wordplay->Python conversion demonstrations (which are longer, and more expensive, given that these are hand-crafted, and selected to demonstrate parts of the DSL in action). Some more detail-oriented work was done on how best to prompt Gemini-Flash to generate code using a doc_string (or otherwise), and whether to prompt for code completion or generation of whole functions, for instance. The final prompt is shown in Appendix A.4.6.
In our comments added directly above, and also in the section for all reviewers (including the updated version of the paper that we uploaded last week), we addressed the following weaknesses and questions that you had : * We explained the structure of cryptic problems more clearly, adding diagrams of 4 of the example problems in (new) Figure 2 (which shows how the wordplay elements in each clue interact to form the final answer) + We also added a reference to Appendix A.1, where there is a longer background description of cryptics with examples of many clue elements * We added motivating language to the paper, which (hopefully) sets the context as being one of looking at reasoning problems more generally (such as mathematics and its related issues with hypothesisation and verification) + Our choice of cryptic crosswords as a reasoning test-bed gives us a different mix of problems to surmount (such as the flexibility of the language used, which leads to verification being a weaker signal). But the fact that Cryptics are less familiar (than (say) maths puzzles or programming challenges) doesn't mean that they don't warrant being taken seriously * Finally, in the updated paper, we added some information about the construction and sensitivity of the ICL prompts, along the lines given above in our previous reply. Is there anything that you need from us to answer any remaining questions that you have? We believe that our new version of the paper substantially tackles all the issues that you identified. We sincerely hope that you can further support our work and would appreciate it if you could let us know whether you are satisfied with our responses to your previous questions.
Summary
The paper proposes a reasoning-based system for solving cryptic crossword clues using fine-tuned LLMs and a Python-based verification process. The model generates candidate answers and wordplay suggestions, formulates these into Python code, and verifies correctness via assertions.
Strengths
**Originality** The paper offers a unique approach to solving cryptic crossword clues. Like math proofs, the authors combined LLMs for candidate generation, wordplay suggestions, and formalized Python-based verifiers. **Quality** The paper details the fine-tuning of models for clue and wordplay generation and the creation of a domain-specific verifier, including formalization in Python. The experiments demonstrate a clear improvement over previous state-of-the-art results on the Cryptonite dataset, significantly outperforming prior baselines. **Clarity** The paper includes figures containing examples that help in understanding. **Significance** The paper shows an improvement in the cryptic crossword domain.
Weaknesses
1. The mention of a 23.5% accuracy on the Cryptonite dataset using GPT-4-turbo in the paper [1] suggests that a more rigorous baseline is possible for this study. Using the same prompting strategy on the Gemini-Flash that is used in the paper could provide better insight into the improvement of the proposed method. 2. The paper's reliance on the Wordplay dataset, gathered from newspapers (The Times, etc.), raises the possibility of data leakage, as The Times also contributes to the Cryptonite test set. If fine-tuning data includes patterns or clues from the test set, results may be artificially inflated. [1] Saha, Soumadeep, et al. "Language Models are Crossword Solvers." arXiv preprint arXiv:2406.09043 (2024).
Questions
1. Partial correctness metrics, as reported in [1], can provide a more nuanced understanding of model success on cryptic crosswords, where the model might have answered some rows/columns but not the whole crossword. 2. Some minor corrections on line 200 ("from" is repeated twice) [1] Saha, Soumadeep, et al. "Language Models are Crossword Solvers." arXiv preprint arXiv:2406.09043 (2024).
Rating
6
Confidence
3
Soundness
3
Presentation
2
Contribution
3
## Re: Weaknesses ### '23.5% GPT-4 accuracy mentioned in Saha (2024)' The result reported in preprint [1] for GPT-4-T with 5-shot prompting on Cryptonite was indeed a surprisingly strong 23.50\% (similarly for their 18.70\% score on the 'Init' version), and we will change ~L174 to reflect this (now that it is clear what '5-shot' means, from their Appendix). We can also include that score 'as reported' in the comparison Table 1 - placing it alongside the Gemini-Pro 1.0 line. However, also in [1] (Table 3), the authors report that giving GPT-4-T instructions to use a Chain-of-Thought approach (with 3-way self-consistency) only resulted in an uplift of +2.15% in the final accuracy on 'Init'. This is a little puzzling, since it indicates that it is likely that GPT-4-T's performance on the test set does not correspond to the kind of deductive process that might be reasonably expected. Considering the typical accuracy uplifts when CoT is used for GSM8K, for instance, models would usually benefit much more from CoT (PaLM improved from 18% to 57% on GSM8K in Wei et al, 2022). Unfortunately, we don't have any visibility into the training set given to GPT-4-T, which apparently stands head-and-shoulders above the other commercial LLMs for this task. As shown in our Table 1, our method does not require a commercial/proprietary LLM to work effectively : Gemini-Flash itself does not add much to the final performance, since it can be replaced with (non-finetuned) Gemma2 9B-it in the Formalisation/coding part (to produce a fully local-LM solution that scores 29.0% on the Cryptonite test set) **EDIT** : We have run the additional baselines as requested - please see the "Up-to-Date Baselines" comment above. ### Reliance on Wordplay Dataset and potential data leakage The Wordplay Dataset splits are specifically chosen to match the train/validation/test splitting mechanism of the Cryptonite dataset. Thus, any clue leading to the answer "EXAMPLE" would only be in one split (the training set in this case) of both the Wordplay and Cryptonite datasets. ## Re: Questions ### Partial Correctness Metrics The partial correctness metrics reported in [1] (Table 1) simulate solving the entire grid. The 'known letters' scores shown in [1] appear to have been obtained by randomly revealing letters within the answers (which is an approximation, since the Cryptonite dataset itself doesn't include information about which letters in an answer are given by cross-words). We agree that this is an interesting dimension to examine for models that are trained to answer the clue directly. However once the models are part of a system, it would only shed light on the first stage in our model's processing - since that is the source of our candidate answers. A natural step here for a *system* would be to iteratively gather the first 20 candidate answers that meet the 'revealed letters' criteria, which is tantamount to using an online crossword solving aid (frowned-upon by human solvers). FWIW, the inclusion of the '70%' column in [1] Table 1 makes no sense in the context of Cryptonite/Init, since the 'Ximenean rules' for crossword grid construction (Macnutt, 1966) mean that grids that appear in papers don't have that density of checking letters. **EDIT** : We have performed some analysis addressing the Partial Correctness Metrics question in an overall Author comment (above), since it is also somewhat related to a point made by another reviewer. ### Refs: * Wei et al, 2022: "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models", https://arxiv.org/abs/2201.11903 * Macnutt, 1966. "Ximenes on the art of the crossword", Published by Methuen, ASIN: B0000CN2M9. (typo fix also implemented - many thanks for catching it)
Thank you for the rebuttal. My concerns are addressed and I have raised my score to 6.
## Up-to-Date Baselines We have run 5-Shot prompting benchmarks to increase the strength of the baseline, and the following results can be substituted for the 'Gemini-1.0-Pro' line in the paper's Table 1: | model | samples | val-overall | val-quick | val-hard | test-overall | test-quick | test-hard | | :---------- | ----------: | ----------: | ----------: | ----------: | ----------: | ----------: | ----------: | | Gemma2-9B-it | 1000 | 5.7% | 11.5% | 5.2% | 4.5% | 10.5% | 4.0% | | Gemini-1.5-Flash-001 | 1000 | 6.6% | 12.5% | 6.1% | 6.5% | 11.8% | 6.1% | | GPT-4o (2024-11-25) | 1000 | 29.8% | 45.0% | 28.5% | 27.6% | 47.4% | 26.0% | Interestingly, this shows that: (a) GPT-4o_2024-11-25 gives stronger results than those of GPT-4-Turbo_2024-04-09 given in Saha (2024); (b) Gemini-Flash (which was used in development of the formaliser) is not particularly good at solving cryptic clues in itself; (c) The Gemma2-9B model gets a large uplift from fine-tuning on the Cryptonite training set (compared to 5-shot prompting). Given the large gap between the performance of GPT-4o and those of Claude 3 Sonnet / GPT 3.5 Turbo reported in Saha (2024), it seems reasonable to postulate that the GPT-4 model has also been fine-tuned on the Cryponite dataset itself. Further evidence being that, as shown in Saha (2024), the GPT-4 model hardly benefits from CoT prompting (with 3-way self-consistency) - it seems that GPT-4 is familiar the facts, but not the reasoning. We will update the paper throughout to reflect the new baseline - against which we note we still have a state-of-the-art result, both with Gemini-Flash as the Formaliser, and (narrowly) with a fully Open-licensed solution (using Gemma2-9B models throughout). An additional aspect to note is that the method provided in the paper provides clearly interpretable output, since the reasoning steps are made directly available (as in Figure 3).
## Exploiting Partially Filled Grids We have run an analysis to address the Exploiting Partially Filled Grids section of Saha et al (2024), the results are as follows (note that these are on the Cryptonite splits, rather than their 'Init', though the percentage differences are large enough to draw some initial conclusions): | model | Hint (%) | samples | val-overall | val-quick | val-hard | test-overall | test-quick | test-hard | | :---------- | ----------: | ----------: | ----------: | ----------: | ----------: | ----------: | ----------: | ----------: | | Gemini-Flash | 25% | 200 | 37.0% | 38.5% | 36.9% | **45.5%** | 66.7% | 43.8% | | Gemma2-9B-it | 25% | 200 | 37.5% | 38.5% | 37.4% | 44.0% | 66.7% | 42.2% | || | FastText k=0 NN | 25% | 200 | 15.5% | 15.4% | 15.5% | 21.0% | 33.3% | 20.0% | | FastText k=0 NN | 50% | 200 | 52.5% | 38.5% | 53.5% | **62.0%** | 46.7% | 63.2% | | FastText k=0 NN | 70% | 200 | 79.0% | 61.5% | 80.2% | **81.0%** | 100.0% | 79.5% | The first two rows show the effect of simply filtering the output of our Gemma2-9B fine-tuned candidate answer proposal model, based on a random letter pattern (using the same formulation as Saha et al (2024)) and then using the rest of our pipeline. Here, our paper's approach outpaces their reported GPT-4T results, but is itself limited by our first stage Gemma2 model's 'Top-20' candidate answers only containing the correct answer only around 45% of the time (as astutely observed by another reviewer). Since the existing Gemma2 has not been fine-tuned on the filtering task, the other top-20 results are essentially restricting our systems performance, even at this low level of partial fill. To gain an insight into higher levels of partial fill, and perhaps bolster our earlier comments on the relevance of these statistics, the 'FastText k=0 NN' lines correspond to running a nearest neighbour (i.e. k=0) search. For each question, we search for the whole clue (without any special processing) over the FastText (Bojanowski et al (2016)) embeddings of a crossword word list (The UK Advanced Cryptics Dictionary - Beresford (2000)) which contains 250,378 valid words that might be found in crosswords (Note that the wordlist is not exhaustive, since 7.0% of the gold answers do not appear in the list). We then use the closest found word as the 'FastText k=0 NN' model answer. While the 25% line is lower-performing than the models considered in our work, the 50% (realistic for solvers) and 70% lines show that even this simplistic nearest neighbour search outperforms all the models of Saha et al (2024). The kNN systematic approach would (naturally) form the baseline for our models in the partially filled grid case, and thus opens up an interesting line of attack for future work (although, as mentioned, we prefer to focus on the stand-alone clue problem, since the reasoning aspect there is most applicable across other fields, such as mathematical theorem proving, where we see no clear analogy with partially filled answers).
### Revised PDF An updated PDF has been uploaded, including many changes that have been prompted by reviewer feedback: * **Motivation** - the initial focus has been altered to emphasise that the core challenge of this work is one of reasoning using LLMs. Cryptic crosswords is a fascinating test-bed for this * **Diagrams** - 4 separate cryptic clues are shown in diagram form (our new Figure 2), which aim to make the principles of cryptic solution clearer for those unfamilar with the topic * **Pipeline Ordering** - an introductory section discusses the choices made, which may have seemed unusual at first * **Wordplay dataset splits** - the deliberate alignment of the Wordplay splits with Cryptonite have been highlighted * **Candidate list size** - this was additionally commented on in the Methods section * **ICL Prompting** - this has been clarified a little, specifically which parts of the process required care, and also the degree of transferability of the ICL prompts from Gemini-Flash directly to Gemma2-9B-it (an off-the-shelf open-licensed model) * **Baselines** - updated baselines have been incorporated into the results. We note that our system still achieves a new state-of-the-art result, and our same pipeline built using fully open-licensed models would also beat the previous state-of-the-art * **Ablations** - 2 ablation results were added, showing the effect of using average-logprob measures from the LLMs generating (a) candidate answers, and (b) wordplay hypotheses. The second of these (which showed limited performance) was a little surprising. Overall, these ablations showed that neither 'dumb ranker' would be able to remove the need for formalisation/verification step. Many thanks are due to reviewer `2vAu` for this insightful suggestion. * **Testing on 'Init'** - we have updated the discussion in Section 2.2 to explain our focus on the Cryptonite dataset, and added a more detailed break-down in the Appendix. The appendix explanation also identifies the looming problem of cross-contamination of training vs test data (specifically for commercial models) between Cryptonite and the Rozner datasets * **Partial Correctness Metrics** - this aspect (more related to solving full grids than the standalone solving that was the main focus of our work), along with a viable solution approach, is now described in our Methods Section 3.6. For reasons of space, the corresponding table of results is in the Appendix. The authors are confident that the paper has been significantly improved through the review process so far, and welcome the opportunity for further discussion that ICLR has provided by extending the author/reviewer discussion period.
We greatly appreciate the reviewers' time and effort in evaluating our paper, including their careful review of our revisions. Your feedback has been invaluable in strengthening our work.
Meta-review
**Summary** The paper fosus on the problem of cryptic crosswords for LLMs by proposing a sort of CoT model to solve varius subtasks. **Strengths** - The paper present a CoT to solve an apparently complex task **Weaknesses** - A solid motivation for why this work is important is missing. Even after the rebuttal, motivation is really weak. The inportance should be better evalated. **Final remarks** - The paper can be better motivated
Additional comments on reviewer discussion
The interaction between the authors and the reviewers is overwhelming. There are too many details to reach the goal of convincing the reviewers to increase their scores. During the discussion period, the authors revealed their identity.