The Theory is Predictive, but is it Complete? An Application to Human Perception of Randomness

When we test a theory using data, it is common to focus on correctness: do\nthe predictions of the theory match what we see in the data? But we also care\nabout completeness: how much of the predictable variation in the data is\ncaptured by the theory? This question is difficult to answer, because in\ngeneral we do not know how much "predictable variation" there is in the\nproblem. In this paper, we consider approaches motivated by machine learning\nalgorithms as a means of constructing a benchmark for the best attainable level\nof prediction.\n We illustrate our methods on the task of predicting human-generated random\nsequences. Relative to an atheoretical machine learning algorithm benchmark, we\nfind that existing behavioral models explain roughly 15 percent of the\npredictable variation in this problem. This fraction is robust across several\nvariations on the problem. We also consider a version of this approach for\nanalyzing field data from domains in which human perception and generation of\nrandomness has been used as a conceptual framework; these include sequential\ndecision-making and repeated zero-sum games. In these domains, our framework\nfor testing the completeness of theories provides a way of assessing their\neffectiveness over different contexts; we find that despite some differences,\nthe existing theories are fairly stable across our field domains in their\nperformance relative to the benchmark. Overall, our results indicate that (i)\nthere is a significant amount of structure in this problem that existing models\nhave yet to capture and (ii) there are rich domains in which machine learning\nmay provide a viable approach to testing completeness.\n