Summary
Motivated by the observation that the posterior variance of a Gaussian process is often poorly calibrated, the authors propose an alternative approach of attaching predictive quantiles to the posterior mean. In essence, their approach minimises the width of the predictive quantiles under an empirical calibration constraint computed on held-out validation data. By satisfying the empirical calibration constraint, the authors prove that the predictive quantiles are indeed approximately the right quantiles (Theorem 5.3). The approach is tested and compared against other calibration approaches in a toy example and on seven data sets from the UCI data set repository. The results show that it outperforms other techniques in terms of sharpness.
Strengths
To begin with, I would like to thank the authors for their submission.
## Strengths
* The paper is easy to follow and generally well written. I found only a few typos.
* The problem that the posterior variance of a GP may be poorly calibrated is highly relevant, so methods that attempt to attack this problem, like the proposed approach by the authors, are certainly important.
* Accompanying the method with a theoretical result that guarantees correctness (Theorem 5.3) reassures the practitioner.
* I really like Section 5.2, where the authors spend some effort on reducing the computational cost in practice. They could have just left it at (7) and state that the optimisation problem would have to be computed for every confidence level $\delta$ of interest, but of course the approach in Section 5.2 is much more elegant.
* The experiments appear to show that the proposed approach is sharper than alternatives. I, however, have some doubts about the experimental section. Please see below.
## General Comments and Suggestions
* Should the title of the paper be "Sharply Calibrated GPs" instead of "Sharp Calibrated GPs"? I think "Sharp(ly)" should be an adverb, since it modifies "calibrated".
* On line 75, you use $\mathcal{D}\_{\text{tr}}$ without introducing the symbol first. This is confusing, because you have just introduced $\mathcal{D}$, not $\mathcal{D}\_{\text{tr}}$, as the training data.
* On line 75, you introduce the cut-point term as $\beta_\delta \sigma_{\mathcal{D}}(\delta, x)$. At this point, you should really better explain this term by answering the following questions: Should $\sigma_{\mathcal{D}}(\delta, x)$ be interpreted as a standard deviation? If so, how do you make sense of that it depends on $\delta$? If not, then what is $\sigma$? If you're learning $\sigma$, why do you also need $\beta_\delta$? Can you not absorb $\beta_\delta$ into $\sigma$? You should also mention that $\beta_\delta$ may be negative. Without that information, (1) doesn't make much sense for small $\delta$.
* On line 81: Do you mean "are small" or "are as small as possible"? This nuance is very important and changes the meaning substantially.
* On line 119, you say that the log-marginal likelihood does not account for calibration. Depending on what precisely you mean by calibration, I think that this is false: the log-marginal likelihood is an empirical estimate of the KL divergence, and the KL divergence certainly accounts for the "whole distribution" and therefore the calibration.
* In Section 5.1, is $\sigma\_{\mathcal{D}\_{\text{tr}}}(\theta, x)$ the posterior variance of the GP where the kernel parameters are now the parameters that we optimise over? From line 146 onwards, you don't actually explain this! Since this is a crucial part of your construction, I think the exposition would be better if you were to clearly explain this somewhere around line 158.
* On line 175, you state that (5) can be replaced by (6) with further explanation. Since this is an important step in the derivation of your algorithm, I think that it deserves a careful explanation. Moreover, it is not true that (5) and (6) are equivalent, since $q\_{\text{lin}}$ is linearly interpolated. I think the exposition would benefit from a little more care here.
* Line 268: "calibration" -> "calibrated"
* Could you add to the legend of Figure 2 what the squares and crosses are? Why does it look like there are two "lines of squares/crossed"?
Weaknesses
## Weaknesses
### Assumption 4.1 Not Obviously Satisfied
I agree that Assumption 4.1 is obviously satisfied for the variance of a kernel. However, for the inverse length scale, I can believe that Assumption 4.1 might be satisfied, but this is not obvious at all. Would the authors be able to produce a proof that the posterior variance of a GP is monotonic in the inverse length scale?
### Theorem 5.3 Might Not Be Valid
The proof of Theorem 5.3 crucially relies on Theorem 1 by Marx et al. (2022). This Theorem 1, however, operates in the setting where all pairs $(x_i, y_i)$ are sampled i.i.d. (see Section 2 of Marx et al., 2022). But in the setting of GPs, which is the setting of the submission, the pairs aren't independent, because they are correlated by the sample from the underlying GP! This means that Theorem 1 might not apply, which means that Theorem 5.3 might not be valid. Could the authors comment on this?
### Result of Section 7.1 Looks Questionable
In Figure 2, you state that your approach produces a 99% confidence interval. However, if you look at the right side of Figure 2.(a), then the confidence interval completely misses the data! It hence looks like the shaded region is not actually a 99% confidence interval, which makes me wonder whether the predictions are actually well calibrated.
### Bolded Results Are Not Significantly Best Results
Throughout the main paper and the supplement, you bold the score with the best average. However, if $x_1 \pm e_1$ and and $x_2 \pm e_2$ are such that $x_1 < x_2$, but $x_1 + e_1 \ge x_2 - e_2$, then you cannot actually conclude with confidence that $x_1 \pm e_1$ is really lower than $x_2 \pm e_2$, because the difference might be explained by random fluctuations. In other words, you should really only bold results that are the best results at reasonable statistical significance. Currently, because of this issue, I think that the results throughout are misrepresented.
### What Are STD and NLL in Table 2.1?
The premise of the paper is that you discard the predictive variance and instead produce calibrated predictive quantiles at one or multiple given confidence levels. This means that the predictions now consist of a mean and associated intervals. Therefore, the predictions are no longer probabilistic, so I really don't understand what the STD and NLL in Table 2 are! (For given a mean and an interval, how can you compute a probability?)
Questions
## Conclusion
Although I think the proposed approach is very interesting, I am mainly worried about the validity of Theorem 5.3, the soundness the result in Section 7.1, the use of boldness to present the experimental results, and the STD and NLL metrics in Table 2.1. Therefore, at this point, I cannot accept this submission and must unfortunately recommend rejection. However, if the authors are able to address the following points, then I am willing to change my reject to an accept:
* Please see if Theorem 5.3 is really flawed, and, if it is, whether it can be fixed.
* Please correct the use of boldness throughout the main paper and the supplement.
* Please investigate whether the predictions in Figure 2.(a) really are calibrated.
* Please explain what the STD and NLL metrics in Table 2.1 are.
EDIT
The authors have largely addressed my criticisms, so I have increased my score to a borderline accept. I believe that the submission might deserve a higher score, but I am not comfortable recommending any higher without seeing a revised version of the PDF.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.