A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering

Legislation can be viewed as a body of prescriptive rules expressed in\nnatural language. The application of legislation to facts of a case we refer to\nas statutory reasoning, where those facts are also expressed in natural\nlanguage. Computational statutory reasoning is distinct from most existing work\nin machine reading, in that much of the information needed for deciding a case\nis declared exactly once (a law), while the information needed in much of\nmachine reading tends to be learned through distributional language statistics.\nTo investigate the performance of natural language understanding approaches on\nstatutory reasoning, we introduce a dataset, together with a legal-domain text\ncorpus. Straightforward application of machine reading models exhibits low\nout-of-the-box performance on our questions, whether or not they have been\nfine-tuned to the legal domain. We contrast this with a hand-constructed\nProlog-based system, designed to fully solve the task. These experiments\nsupport a discussion of the challenges facing statutory reasoning moving\nforward, which we argue is an interesting real-world task that can motivate the\ndevelopment of models able to utilize prescriptive rules specified in natural\nlanguage.\n