Rissanen Data Analysis: Examining Dataset Characteristics via Description Length

We introduce a method to determine if a certain capability helps to achieve\nan accurate model of given data. We view labels as being generated from the\ninputs by a program composed of subroutines with different capabilities, and we\nposit that a subroutine is useful if and only if the minimal program that\ninvokes it is shorter than the one that does not. Since minimum program length\nis uncomputable, we instead estimate the labels' minimum description length\n(MDL) as a proxy, giving us a theoretically-grounded method for analyzing\ndataset characteristics. We call the method Rissanen Data Analysis (RDA) after\nthe father of MDL, and we showcase its applicability on a wide variety of\nsettings in NLP, ranging from evaluating the utility of generating subquestions\nbefore answering a question, to analyzing the value of rationales and\nexplanations, to investigating the importance of different parts of speech, and\nuncovering dataset gender bias.\n

Paper

Similar papers

© 2026 NYSGPT2525 LLC