This position paper describes and critiques the Pretraining-Agnostic\nIdentically Distributed (PAID) evaluation paradigm, which has become a central\ntool for measuring progress in natural language understanding. This paradigm\nconsists of three stages: (1) pre-training of a word prediction model on a\ncorpus of arbitrary size; (2) fine-tuning (transfer learning) on a training set\nrepresenting a classification task; (3) evaluation on a test set drawn from the\nsame distribution as that training set. This paradigm favors simple, low-bias\narchitectures, which, first, can be scaled to process vast amounts of data, and\nsecond, can capture the fine-grained statistical properties of a particular\ndata set, regardless of whether those properties are likely to generalize to\nexamples of the task outside the data set. This contrasts with humans, who\nlearn language from several orders of magnitude less data than the systems\nfavored by this evaluation paradigm, and generalize to new tasks in a\nconsistent way. We advocate for supplementing or replacing PAID with paradigms\nthat reward architectures that generalize as quickly and robustly as humans.\n