Summary
Background:
Generating reliable probability estimates alongside accurate class labels is crucial in classification tasks. Calibration, which refers to the alignment between predicted probabilities and empirical frequencies of labels, is highly desirable in various applications. However, many machine learning algorithms lack inherent calibration.
This paper aims to address these issues in two ways:
1. Development of a unified framework for recalibration that incorporates both calibration and sharpness in a principled manner.
2. Proposal of a composite estimator for recalibration under label shift, which converges to the optimal recalibration and enables sample-efficient adaptation of classifiers to label-shifted domains.
Strengths
1. This paper is exceptionally well-written, with a clear and logical structure, making it easily understandable to a wide audience.
2. The theoretical foundation of this paper is solid, the proof process is very detailed, making it easy for people to follow.
3. The algorithm proposed by the author is simple and efficient, making it easy to implement.
Weaknesses
1. Although the author provides solid theoretical analysis, this paper lacks enough experiments to demonstrate the effectiveness of their method.
2. The author employs three assumptions, and in my view, assumption 2 is a rather strong one, which is difficult to guarantee in practical applications.
3. The experiments in this paper were conducted primarily on toy datasets, and the author do not provide any experiments on how to apply Recalibration in real-world applications.
Questions
1. In Eq. (9), I understand that the function $\hat{h}(z)$ calculates the expected label within the bin that $z$ belongs to. So, why is $\hat{h}$ a monotonically increasing function of $z$ and its growth pattern similar to the cumulative distribution function (CDF) of $z$? Although you have made Assumption 2, can I understand that this assumption implies that $y$ must follow a specific distribution form in order to satisfy the monotonicity of $\hat{h}$?
2. In the experimental section, the author only presents a very simple toy scenario, and I am quite curious about the performance of the author's algorithm in more complex scenarios. For instance, in real-world classification datasets like cifar-10-long-tail or cifar-100-long-tail, how does the recalibration effect of more complex models, such as neural networks or decision trees?
3. In Fig. 1 (a), as the value of $n$ increases, regardless of the magnitude of $B$, $R^{cal}(\hat{h})$ exhibits a noticeable decrease. Does this imply that when the dataset size $n$ becomes sufficiently large, $R^{cal}(\hat{h})$ has already reached a level where recalibration is unnecessary?
If the author can answer my question, I will change my score.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.