A trained ML model is deployed on another `test' dataset where target feature\nvalues (labels) are unknown. Drift is distribution change between the training\nand deployment data, which is concerning if model performance changes. For a\ncat/dog image classifier, for instance, drift during deployment could be rabbit\nimages (new class) or cat/dog images with changed characteristics (change in\ndistribution). We wish to detect these changes but can't measure accuracy\nwithout deployment data labels. We instead detect drift indirectly by\nnonparametrically testing the distribution of model prediction confidence for\nchanges. This generalizes our method and sidesteps domain-specific feature\nrepresentation.\n We address important statistical issues, particularly Type-1 error control in\nsequential testing, using Change Point Models (CPMs; see Adams and Ross 2012).\nWe also use nonparametric outlier methods to show the user suspicious\nobservations for model diagnosis, since the before/after change confidence\ndistributions overlap significantly. In experiments to demonstrate robustness,\nwe train on a subset of MNIST digit classes, then insert drift (e.g., unseen\ndigit class) in deployment data in various settings (gradual/sudden changes in\nthe drift proportion). A novel loss function is introduced to compare the\nperformance (detection delay, Type-1 and 2 errors) of a drift detector under\ndifferent levels of drift class contamination.\n