Tracking the risk of a deployed model and detecting harmful distribution shifts

When deployed in the real world, machine learning models inevitably encounter\nchanges in the data distribution, and certain -- but not all -- distribution\nshifts could result in significant performance degradation. In practice, it may\nmake sense to ignore benign shifts, under which the performance of a deployed\nmodel does not degrade substantially, making interventions by a human expert\n(or model retraining) unnecessary. While several works have developed tests for\ndistribution shifts, these typically either use non-sequential methods, or\ndetect arbitrary shifts (benign or harmful), or both. We argue that a sensible\nmethod for firing off a warning has to both (a) detect harmful shifts while\nignoring benign ones, and (b) allow continuous monitoring of model performance\nwithout increasing the false alarm rate. In this work, we design simple\nsequential tools for testing if the difference between source (training) and\ntarget (test) distributions leads to a significant increase in a risk function\nof interest, like accuracy or calibration. Recent advances in constructing\ntime-uniform confidence sequences allow efficient aggregation of statistical\nevidence accumulated during the tracking process. The designed framework is\napplicable in settings where (some) true labels are revealed after the\nprediction is performed, or when batches of labels become available in a\ndelayed fashion. We demonstrate the efficacy of the proposed framework through\nan extensive empirical study on a collection of simulated and real datasets.\n