Summary
This paper analyzes the estimation bias and generalization error of the expected calibration error (ECE). Specifically, in a binary classification setting, the authors provide an upper bound for the total bias with an improved convergence rate, applicable to both uniform mass and uniform width binning strategies. They also determine the optimal number of bins to minimize the total bias. Furthermore, the authors utilize the information-theoretic generalization framework, particularly the Conditional Mutual Information (CMI) framework, to characterize the generalization of ECE.
Strengths
1. This paper achieves a tighter bound for total bias compared to previous works.
2. The optimal number of bins is determined using the upper bound of the total bias.
Weaknesses
1. As the authors themselves note, a significant limitation is that the analysis in this work is only applicable to binary classification.
2. Some assumptions (e.g., Assumption 2) are not well justified.
3. The writing has significant room for improvement; several arguments are unclear or misleading.
Please find more details in the questions below.
Questions
1. Does Assumption 2 hold true in practice? Is there a way to verify it? Additionally, what is the motivation behind assuming $n_e\geq 2B$? How is this assumption utilized? If $n_{te}\leq 2B$, will Theorems 2 and 3 still be valid?
2. In the proof sketch of Theorem 2, you mention that $\mathrm{ECE}$ and $\mathrm{TCE}$ could be re-written. While they seem correct to me, could you elaborate on how $\mathrm{TCE}(f_\mathcal{I})$ is obtained in the form shown in Line 152? I did not find the details in the complete proof.
3. According to Theorem 5, do the upper bounds indicate that UWB is a better binning strategy than UMB, given that UMB has an additional $\mathrm{fCMI}$ bias term? It seems that only UMB's expected binning bias is sensitive to the training data, which might be seen as a disadvantage in terms of the upper bound.
4. The writing can be significantly improved. For example, in Line 244, you mention "Our theory might guarantee the ECE under test dataset for them." Do you mean your theory might guarantee low ECE under the test dataset? Additionally, in Lines 251-252, "This implies that if the model generalizes well, evaluating the ECE using the training dataset may better reduce the total bias than that using test dataset." Why does evaluating ECE reduce total bias? What we really care about is ECE on unseen/test data. How does evaluating ECE on training data affect this purpose?
5. Why is the metric entropy method only used for UWB? It seems that you upper bound $\mathrm{eCMI}$ by the $\mathrm{fCMI}$ term first in your proof. What prevents you from giving a similar result for UMB?
6. In Line 339-340, you mention that "a notable trend towards acquiring relatively stable nonvacuous bounds can be observed when adopting $B =\lfloor n^{1/3} \rfloor$", but according to Figure 1, it seems $B=52$ is tighter than $B=B =\lfloor n^{1/3} \rfloor$ in most cases. Could you clarify this?
7. Since $\mathrm{eCMI}$ and $\mathrm{fCMI}$ terms are key components in both standard generalization error and expected total bias of calibration error, do you have any new insights into the relationship between calibration and generalization from this perspective?
Limitations
The authors have thoroughly discussed some limitations of this work.