Calibration ensures that probabilistic forecasts meaningfully capture uncertainty by requiring that predicted probabilities align with empirical frequencies. However, many existing calibration methods are specialized for post-hoc recalibration, which can worsen the sharpness of forecasts. Drawing on the insight that calibration can be viewed as a distribution matching task, we introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression. These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization. Furthermore, we provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions. Our empirical evaluation demonstrates that employing these metrics as regularizers enhances calibration, sharpness, and decision-making across a range of regression and classification tasks, outperforming methods relying solely on post-hoc recalibration.
Paper
Similar papers
Peer review
Summary
Accurate probability prediction is a crucial aspect of a trustworthy model, especially for deep neural networks. This paper focuses on the model calibration problem and proposes a unified framework via distribution matching. To leverage the distribution matching, they used Maximum Mean Discrepancy to estimate the miscalibration, where MMD is a popular distribution measure and used in many distribution alignment problems in recent years. The main usage of MMD in this paper is to treat it as a trainable regularizer. In the experiment section, they demonstrate their effectiveness in performing proper distribution alignments.
Strengths
1. They introduced a unified framework for calibration metrics as distribution-matching problems, which is a very interesting view for uncertainty calibration. 2. They proposed a training-time model calibration method by introducing kernel-based calibration metrics.
Weaknesses
The paper does not have main weakness, if some, they are: 1. How to apply this method to multi-class classification? I did not see the related experiments. It seems that all the used datasets are regression datasets, but Line 6 said it is also suitable for classification tasks. 2. Deep comparisons with similar kernel methods, like [1]. I did not see the description of their difference. They also used the kernel method to perform model calibration. [1] Trainable Calibration Measures For Neural Networks From Kernel Mean Embeddings. ICML 2018 3. What are the n and d in Table 2? Why do you use the different n and d regarding different datasets? 4. How to demonstrate the prediction performance improvement using the proposed methods? I did not find the clue in Table 2.
Questions
Please refer to the weaknesses.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Yes
Summary
This paper presents a general framework for train-time calibration based on kernel MMD. The authors demonstrate empirically that choosing good kernels (based on the task at hand) in their framework can lead to improvements in calibration while maintaining sharpness of predictions. Update: I have increased my score to a 6 in light of the authors' responses. I think the empirical portions of the paper could still be improved (ideally comparing to more training-time calibration methods), but I believe the sharpness improvements over MMCE are interesting and useful.
Strengths
- **Originality:** While kernel-based calibration methods have been studied in the literature, the perspective provided in this work is original in that the authors show how various different types of calibration can be captured by their MMD framework. - **Quality:** The quality of the work is overall good, with appropriate support and examples provided for the claims in the paper. - **Clarity:** The paper is mostly well-written. However, there are implementation details regarding the MMD approach that I feel are important and missing (detailed further in weaknesses). - **Significance:** This paper adds to the growing body of work showing the benefits of kernel-based methods for improving calibration. However, it is difficult to assess the relative significance of this work (compared to, for example [1]) due to the limited experiments. [1] https://proceedings.mlr.press/v80/kumar18a.html
Weaknesses
- **Motivation for Implementation Details:** It would be very helpful to accompany the discussion in Section 4 with an idea of how the conditional kernel MMD is implemented in practice and why a practitioner would make certain decisions. For example, is using a model parameterization like the one in the paper actually that necessary (i.e. predicting parameters of a Gaussian)? Can one not just take a similar approach to [1]? - **Experiments:** The experiments in the paper compare to a baseline of training using NLL (and post-hoc calibration), with no comparisons to other train-time calibration approaches. As a result, it is difficult to assess the relative worth of the proposed methodology compared to existing approaches. - **Contextualization:** The MMD framework proposed in this paper is very similar to the ideas in [1], and I think the paper would benefit from more direct comparisons to [1] and related prior work that highlight the contributions of this work. To summarize, I feel that the lack of implementation details + lack of comparison to other train-time calibration approaches limit the contribution of the paper.
Questions
- How much longer does it take to train (wall clock time) using the MMD regularization? - How important is the choice of output distribution for the model considered? In other words, did you consider predicting the parameters of a distribution other than Gaussian and comparing? - As asked in the weaknesses, can we relax the need to differentiably sample from the predictive distribution? - In the supplementary material, I believe under distribution calibration it should be $P_{Y \mid X}(y) \mid Q_{Y \mid X} = Q_{Y \mid X}(y)$.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
While the authors do discuss limitations, I feel the limitations mentioned should have received further exposition in the main body of the work.
Summary
This paper introduces differentiable kernel-based calibration metrics that unify and generalize popular forms of calibration. The notions of calibration are presented as distribution matching constraints by using various kernels in the Maximum Mean Discrepancy (MMD) metric. Through empirical evaluation, it is demonstrated that incorporating these metrics as regularizers enhances calibration, sharpness and facilitates better decision-making in regression scenarios.
Strengths
- The proposed metrics can be integrated in gradient based optimization alongside proper scoring rules. - The paper extends on the discussion regarding the connection between calibration and decision making, which offers valuable insights for practical problems. - The empirical results are sufficiently extensive and verify the effectiveness of integrating the proposed metrics into training objectives, as a means to mitigate calibration error.
Weaknesses
The related work section does not sufficiently discuss prior works, and fails to highlight the specific advancements made by this paper in comparison. As the current paper proposes a kernel-based differentiable metric to be used as a regularizer alongside a proper scoring rule, there has to be a discussion of existing trainable calibration methods, especially kernel-based methods like [Kumar et al., 2018, Zhang et al., 2020, Popordanoska et al., 2022]. *[Zhang et al., 2020] Jize Zhang, Bhavya Kailkhura, and T Han. Mix-n-Match: Ensemble and compositional methods for uncertainty calibration in deep learning. In ICML 2020* *[Popordanoska et al., 2022] Teodora Popordanoska, Raphael Sayer, Matthew B. Blaschko A Consistent and Differentiable Lp Canonical Calibration Error Estimator. In NeurIPS 2022*
Questions
- The abstract claims that the proposed metric generalizes popular forms of calibration for classification and regression. Could you clarify the relationship of the metric with notions of calibration used in classification, i.e. top-label, marginal and canonical (see e.g. Eq. 1 2 and 3 in [Vaicenavicius et al., 2019, Kull et al., 2019]? - What is the computational overhead of the proposed method? - How is the bandwidth of the kernels chosen? *[Vaicenavicius et al. 2019] Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, Thomas B. Schön Evaluating model calibration in classification. In AISTATS 2019.* *[Kull et al. 2019] Meelis Kull, Miquel Perello-Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, Peter Flach Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration. In NeurIPS 2019*
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Limitations are discussed in section 7.
Summary
Calibration as a training regularizer: This paper focuses on investigating how to incorporate calibration metrics as training regularizers. It starts with summarizing different calibration metrics using a unified view from distribution matching. The paper then proposed a kernel-based general metric that maps labels and conditioning variables to an RKHS space and measures the maximum mean discrepancy on the kernel space. The paper then extends the metric to the decision-making space with two concrete examples. In experiments, combined with post-hoc calibration, using the proposed calibration kernel metric as a trainable regularizer can achieve a better evaluation of calibration in regression problems.
Strengths
Pros: The paper provides a unified view of many different calibration metrics. The kernel-based calibration regularizer is novel. The paper is relatively clearly written.
Weaknesses
Cons: 1. The decision calibration seems to be a natural extension of the section 4, as the decision calibration framework has been proposed before. So the main contribution is based on the kernel-MMD-based calibration metric, which is relatively straightforward. 2. Also, the paper only suggests using the metric as a regularizer to incentivize decision calibration during training but does not really guarantee anything or analyze formally the impact on the decision calibration scenario. 3. The trainable regularizer needs to be combined with post-hoc calibration method, which may raise the question how effective the method is. Table 2 shows that the MMD alone does not always provide enough improvement.
Questions
I may have missed it, but I am a little confused by the correspondence between the trainable calibration metric and the evaluation. Table 2 shows training using NLL and the proposed method (individual calibration) and evaluating using both quantile calibration and decision calibration. Why quantile calibration is used here? Why not individual calibration in the evaluation? What happens if the problem is a classification problem? The evaluation of calibration is classification is usually tricky. How much the trainable regularize can benefit there? Minor: table caption is usually on top of the table. ===== After reading the rebuttal, I raised my score to 5 but still concerned about the lack of theoretical guarantees and the requirement of additional calibration data and step.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
2 fair
Limitations
The paper discussed technical limitations.
After reading the rebuttal
Thanks for responding to my questions. Most of my concerns are addressed. I am happy to keep my score.
Thank you for the clarifications and the comparison to MMCE. For the comparison to MMCE, what exactly was the setup (i.e. did you follow their settings or use the same kernels as your models)? There seems to be some strange (possibly anomalous) behavior on the heart-disease dataset in which training with MMCE reduced accuracy and calibration significantly, which is surprising - everywhere else the performance of MMCE essentially matches that of the proposed method (the biggest ECE improvement is about 0.02, and this may not even be maintained across different ECE estimation procedures). My main question remains regarding the actual implementation details of the approach outlined in the paper - is a correct understanding that one chooses a kernel and a forecast distribution and then approximates the expectation in the MMD via sampling? If so, how many samples are used? I think in revising the paper it would be helpful to clearly outline the exact implementation details a practitioner would need to follow to implement the approach in practice.
Thank you for engaging with our work! We truly believe that your comments and suggestions have helped us in strengthening our work and clarifying the experimental setup. We address your follow-up comments below. > **For the comparison to MMCE, what exactly was the setup (i.e. did you follow their settings or use the same kernels as your models)?** We adopted the same setup for MMCE as detailed in [Kumar et al., 2018]. Specifically, we utilized a Laplacian Kernel with a width of 0.4, and the $\lambda$ parameter for weighting MMCE with respect to NLL was chosen through a hyperparameter sweep. For MMD metrics, we followed the exact same setup detailed in Section 6.2 of our paper. For both MMCE and our models, we perform a hyperparameter sweep and select the best performing model configurations based on model accuracy on a held-out validation set. We retrain the best model configurations using 50 random seeds to obtain the final results presented in our tables. > **There seems to be some strange (possibly anomalous) behavior on the heart-disease dataset in which training with MMCE reduced accuracy and calibration significantly, which is surprising.** We ran additional hyperparameter sweeps to verify the results on the heart-disease dataset, and found the results to be consistent across runs. On the datasets we tested, we found that MMCE typically improved ECE and worsened entropy, relative to training with only NLL. We note that prior work has also found that MMCE worsens ECE and accuracy on some datasets. For example, on ResNet-110 [Mukhoti et al., 2021, Table 1] reports for MMCE that ECE=5.51 and error=5.4, as opposed to ECE=4.41 and error=4.89 when training with only cross-entropy. To further facilitate the reproducibility of our results, we plan to release our code. > **The performance of MMCE essentially matches that of the proposed method.** We would like to highlight that the main motivation for calibration regularizers is to mitigate the tradeoff between calibration and sharpness. We find that our method yields better sharpness than MMCE on every dataset we tested, often by a significant margin. MMCE gives entropy that ranges from 1.45x to 5.08x greater than ours w/o temp scaling, and 1.02x to 1.38x greater than ours w/ temp scaling (see Tables 1 & 2 in the pdf attached to our response). Essentially, compared to MMCE, our method achieves comparable (and often better) calibration without degrading sharpness. > **Is a correct understanding that one chooses a kernel and a forecast distribution and then approximates the expectation in the MMD via sampling? If so, how many samples are used?** Yes, your understanding is correct. To be clear, the metric is specified by a choice of kernel, target variable, forecast variable, and conditioning variable. To estimate the metric, we use sample estimates based on the dataset (e.g. ground truth $X$, $Y$) , and samples from the model’s forecasted distribution (e.g. sampled from $Q_{Y|X}$). For classification, we provide the expression for our estimator in the general response above (no sampling from the forecast is required). For the regression setting, we discuss the estimator of MMD in lines 124-128 of our paper. Although we mention that marginalizing randomness of the forecaster improves training stability in practice, we agree that this point can be easily missed, and will make sure to clarify it further in the revised paper. We found that increasing the number of samples from the forecasted distribution improves stability at the cost of additional compute. See the below table for quantitative results on the effect of the number of forecast samples on the NLL, QCE, and DCE. We find that model performance stabilizes at around 10 forecast samples per data example. This was the setting used in our experiments, and reflects the 1.2x wall clock training time relative to training with only NLL. Table values follow the format of Negative Log Likelihood (NLL) / Quantile Calibration Error (QCE) / Decision Calibration Error (DCE). All model hyperparameters were fixed between runs. | # samples | crime | blog | medical-expenditure | superconductivity | fb-comment | |:---:|:---:|:---:|:---:|:---:|:---:| | 1 | -0.703/ 0.198/ 0.088 | 0.952/ 0.379/ 4.171 | 1.546/ 0.071/ 0.448 | 3.369/ 0.053/ 0.182 | 0.637/ 0.268/ 3.150 | | 5 | -0.720/ 0.154/ 0.043 | 0.96/ 0.395/ 4.090 | 1.539/ 0.071/ 0.419 | **3.311**/ 0.038/ 0.216 | **0.409**/ 0.276/ 3.146 | | 10 | -0.777/ 0.154/ 0.060 | 0.85/ 0.416/ 4.077 | 1.54/ 0.072/ 0.459 | 3.333/ 0.036/ 0.208 | 0.590/ 0.278/ **3.102** | | 50 | -0.779/ **0.15**/ **0.041** | 0.847/ 0.432/ 4.016 | **1.531**/ 0.065/ 0.447 | 3.345/ 0.041/ 0.208 | 0.479/ **0.223**/ 3.140 | | 100 | -0.781/ 0.153/ 0.043 | **0.829**/ 0.386/ **3.978** | 1.536/ **0.064**/ 0.449 | 3.318/ 0.036/ **0.181** | 0.469/ 0.230/ 3.149 | | 200 | **-0.782**/ 0.151/ 0.043 | **0.829**/ **0.373**/ **3.978** | 1.535/ **0.064**/ 0.442 | 3.291/ **0.034**/ 0.182 | 0.458/ 0.231/ 3.137 |
Thank you very much for the further clarifications; I had glossed over the improvements in NLL relative to MMCE when I looked at the updated results. This is indeed a useful attribute of the proposed method, and I think in revising it would be helpful to emphasize this further (possibly with additional visualizations/investigations into sharpness vs calibration in the appendix). My only remaining question is regarding the NLL performance on the crime dataset - how is it negative? Sorry I missed this when doing an initial pass over the paper.
We are glad you appreciate the contributions of our method for simultaneously achieving calibration and sharpness. Thank you for your suggestions! We will clarify and expand on the value of preserving sharpness in the revised paper. > My only remaining question is regarding the NLL performance on the crime dataset - how is it negative? The negative log-likelihood can be negative in a regression setting, since the probability density function can take values greater than one. For a concrete example, consider a forecaster $q$ that, given features $x$, outputs a Gaussian distribution $q_{Y \mid x} = \mathcal{N}(y; \mu(x), \sigma)$ with mean $\mu(x)$ and a fixed standard deviation $\sigma$. Suppose that the true label follows the forecasted Gaussian distribution $Y \sim q_{Y \mid x}$. The negative log-likelihood (NLL) is given by: $$ NLL(D, q) = \mathbb{E}_ {(x,y) \sim D} \left[ - \log q_{Y \mid x} (y) \right] $$ Then the NLL simplifies to: $$ NLL(D, q) = \log \left( \sqrt{2 \pi \sigma^2} \right) + \frac{1}{2} $$ For a small standard deviation $\sigma = 0.1$, the above equation evaluates to the negative value $NLL(D, q) = -0.884$.
Ah right of course, my apologies for missing that. I've upgraded my score to a 6. Thank you for the responses!
Post rebuttal
I thank the authors for answering my questions and addressing my concerns. I would like to keep my original score.
Decision
Accept (poster)