Distribution-free uncertainty quantification for classification under label shift

Trustworthy deployment of ML models requires a proper measure of uncertainty,\nespecially in safety-critical applications. We focus on uncertainty\nquantification (UQ) for classification problems via two avenues -- prediction\nsets using conformal prediction and calibration of probabilistic predictors by\npost-hoc binning -- since these possess distribution-free guarantees for i.i.d.\ndata. Two common ways of generalizing beyond the i.i.d. setting include\nhandling covariate and label shift. Within the context of distribution-free UQ,\nthe former has already received attention, but not the latter. It is known that\nlabel shift hurts prediction, and we first argue that it also hurts UQ, by\nshowing degradation in coverage and calibration. Piggybacking on recent\nprogress in addressing label shift (for better prediction), we examine the\nright way to achieve UQ by reweighting the aforementioned conformal and\ncalibration procedures whenever some unlabeled data from the target\ndistribution is available. We examine these techniques theoretically in a\ndistribution-free framework and demonstrate their excellent practical\nperformance.\n