Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning

This paper studies the problem of post-hoc calibration of machine learning\nclassifiers. We introduce the following desiderata for uncertainty calibration:\n(a) accuracy-preserving, (b) data-efficient, and (c) high expressive power. We\nshow that none of the existing methods satisfy all three requirements, and\ndemonstrate how Mix-n-Match calibration strategies (i.e., ensemble and\ncomposition) can help achieve remarkably better data-efficiency and expressive\npower while provably maintaining the classification accuracy of the original\nclassifier. Mix-n-Match strategies are generic in the sense that they can be\nused to improve the performance of any off-the-shelf calibrator. We also reveal\npotential issues in standard evaluation practices. Popular approaches (e.g.,\nhistogram-based expected calibration error (ECE)) may provide misleading\nresults especially in small-data regime. Therefore, we propose an alternative\ndata-efficient kernel density-based estimator for a reliable evaluation of the\ncalibration performance and prove its asymptotically unbiasedness and\nconsistency. Our approaches outperform state-of-the-art solutions on both the\ncalibration as well as the evaluation tasks in most of the experimental\nsettings. Our codes are available at\nhttps://github.com/zhang64-llnl/Mix-n-Match-Calibration.\n