Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Language

Current NLP datasets targeting ambiguity can be solved by a native speaker\nwith relative ease. We present Cryptonite, a large-scale dataset based on\ncryptic crosswords, which is both linguistically complex and naturally sourced.\nEach example in Cryptonite is a cryptic clue, a short phrase or sentence with a\nmisleading surface reading, whose solving requires disambiguating semantic,\nsyntactic, and phonetic wordplays, as well as world knowledge. Cryptic clues\npose a challenge even for experienced solvers, though top-tier experts can\nsolve them with almost 100% accuracy. Cryptonite is a challenging task for\ncurrent models; fine-tuning T5-Large on 470k cryptic clues achieves only 7.6%\naccuracy, on par with the accuracy of a rule-based clue solver (8.6%).\n