Reply (part 1/2)
Thank you for your detailed review and suggestions. Below, we address each of your points (in the same order as the original bullet points).
## Theory/method questions:
* Regarding distinction between $\widetilde{SmECE}$ and $SmECE$: we agree that indeed $SmECE$ is a slightly more natural notion as a calibration measure. Those two notions are close to each other, both in spirit, as well as formally (see for instance Lemma 10 in appendix A). On the other hand, the $\widetilde{SmECE}$ is more directly reflected in a visually interpretable reliability diagram, which we argue for constructing. As such, we chose to define both those notions and highlight the close relations between those two, while arguing for SmECE to be the more fundamental construction, which should be eventually used to report calibration error itself. A slightly more detailed discussion about this is present in Appendix A.
* Choosing the fixed point $\sigma^*$ is to a small extent arbitrary. Lemmas 7 and 8 provide an upper and lower bounds of the distance to the nearest perfectly calibrated predictor in terms of $smECE_\sigma$ and $\sigma$. This leads to a range of parameters $\sigma$ for which those upper and lower bounds are good enough for the measure to satisfy the “consistency” requirement. One could, for instance, choose $\mu(D) := \min_{\sigma} (smECE_{\sigma}(D) + \sigma)$. Such a $\mu$ would also be a consistent calibration measure. Yet it wouldn’t necessarily provide a specific scale $\sigma$ for which $smECE_{\sigma}(D)$ is proportional to $\mu(D)$ — and the quantity related with $smECE_{sigma}$ is the one we are presenting on the reliability diagram. On the other hand, choosing a fixed point $\sigma_*$, provides an approximate minimizer for $smECE_{\sigma}(D) + \sigma$ (up to a factor of two, since $smECE_{\sigma}(D)$ is non-increasing with sigma), provides a scale on which $smECE_{\sigma}(D)$ (which is visualized on the diagram) is proportional to $smECE_{\sigma}(D) + \sigma$ (which is guaranteed to upper bound the distance to calibration), is easy to compute, and is in a sense “canonical” in a range of parameters sigma for which we would have the consistency property — a choice that can be universally agreed on, to end up with a parameter-free calibration measure with respect to which different classifiers can be easily compared.
* Thank you for pointing out that the importance of consistent calibration measures was not discussed enough, that is a great point. We have included additional discussion in the current write-up (Appendix E), which we will consider moving into the main body in the full version (where we will have more space).
* The more precise upper and lower bounds between smECE and distance to the nearest perfectly calibrated predictor are given at the end of Section 3.3 (giving $\alpha_1 = 1, \alpha_2=1/2$). This provides a decent continuity guarantee for the smECE measure. We provided some synthetic examples in Figure 3, where this continuity is highlighted — binning ECE and the associated binning diagram behaves extremely differently on two very close datasets of (prediction, outcome), while smECE provides a stable picture of mis-calibration.
* Regarding intuition behind Lemma 7 and Lemma 8. Lemma 7 is best understood in terms of duality theorem for distance to calibration: such a distance is (up to a constant factor) equal to maximum over all Lipschitz functions $w: [0,1] \to [-1,1]$ of a correlation between $w$ applied to the prediction $f$, and the residual between prediction $f$ and outcome $y$,
i.e. $\sup_{w} \mathbb{E}[w(f) (f-y)] $
(in this paper this latter quantity is called weak calibration error, see Definition 19 and Theorem 20 in Appendix D.1). With this in hand, it’s not that surprising why quantities like $(\widetilde{smECE}_\sigma + \sigma)$ upper bound the weak calibration error. For any given Lipschitz function $w$, we can bound the $\mathbb{E}[w(f)(f-y)]$ in terms of $ECE(\pi_R(f+Z), y)$ and $\sigma$ where $Z$ is a an independent gaussian with standard deviation sigma. Indeed, while extending the notion of smECE to non-trivial metrics on the space of predictions $[0,1]$, one of the main steps we had to take care of was generalization of the duality theorem (see Appendix B.1 and D.8). Lemma 8 follows from observing that smECE(D) is Lipshitz-continuous with respect to the Wasserstein metric on distribution D (see Lemma 23 for a formal statement of this). Then, by comparing smECE(D) to smECE(D'), where D' is the closest perfectly-calibrated distribution to D, we obtain the bound in Lemma 8.