Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensembles

When a deep learning model is deployed in the wild, it can encounter test\ndata drawn from distributions different from the training data distribution and\nsuffer drop in performance. For safe deployment, it is essential to estimate\nthe accuracy of the pre-trained model on the test data. However, the labels for\nthe test inputs are usually not immediately available in practice, and\nobtaining them can be expensive. This observation leads to two challenging\ntasks: (1) unsupervised accuracy estimation, which aims to estimate the\naccuracy of a pre-trained classifier on a set of unlabeled test inputs; (2)\nerror detection, which aims to identify mis-classified test inputs. In this\npaper, we propose a principled and practically effective framework that\nsimultaneously addresses the two tasks. The proposed framework iteratively\nlearns an ensemble of models to identify mis-classified data points and\nperforms self-training to improve the ensemble with the identified points.\nTheoretical analysis demonstrates that our framework enjoys provable guarantees\nfor both accuracy estimation and error detection under mild conditions readily\nsatisfied by practical deep learning models. Along with the framework, we\nproposed and experimented with two instantiations and achieved state-of-the-art\nresults on 59 tasks. For example, on iWildCam, one instantiation reduces the\nestimation error for unsupervised accuracy estimation by at least 70% and\nimproves the F1 score for error detection by at least 4.7% compared to existing\nmethods.\n