In the 1980s a new, extraordinarily productive way of reasoning about\nalgorithms emerged. In this paper, we introduce the term "outcome reasoning" to\nrefer to this form of reasoning. Though outcome reasoning has come to dominate\nareas of data science, it has been under-discussed and its impact\nunder-appreciated. For example, outcome reasoning is the primary way we reason\nabout whether ``black box'' algorithms are performing well. In this paper we\nanalyze outcome reasoning's most common form (i.e., as "the common task\nframework") and its limitations. We discuss why a large class of\nprediction-problems are inappropriate for outcome reasoning. As an example, we\nfind the common task framework does not provide a foundation for the deployment\nof an algorithm in a real world situation. Building off of its core features,\nwe identify a class of problems where this new form of reasoning can be used in\ndeployment. We purposefully develop a novel framework so both technical and\nnon-technical people can discuss and identify key features of their prediction\nproblem and whether or not it is suitable for outcome reasoning.\n