Response
We thank the reviewer for their comments.
> The abstract promises "improve[d] predictive accuracy, calibration, and out of distribution detection over baselines.", similarly in the contributions, and conclusion parts of the paper. Even more, the method not only improves, but it also performs "extremely strong" and "exceptionally strong". These are some exceptionally strong statements given the actual performance.
Focusing on each of the experiments in turn. The first problem is the presentation, e.g., what does a bold number mean (see question below)?
We understand that you feel the claimed contributions of the method are stated too strongly. We have revised our stated contributions to be more moderate.
> Regression Experiments. Of the six data sets (see question below on this number) the proposal improves on two, slightly on one, equally on two (although better than neural net-based baselines), and worse than most of its baselines in the final one. Calling this "strong performance" is rather misleading. Two additional, though potentially minor, problems are that all of the baselines are simply cited from prior work (Watson et al., 2021). Given the wide performance variations between different train/test splits that can be observed for various UCI data sets the results are not entirely trustworthy. (Note that the reported error intervals are most likely standard errors, as is common on UCI, instead of standard deviations. But which they are is never specified.) Secondly, the authors acknowledge in the appendix that there might be differences in the way the training data is normalized compared to the cited results. (Whether these problems strongly influence the results, or bias them in favor or against VBLL is unclear.)
We partially disagree with the reviewer about the strength of the method on regression experiments. The cases where our approach outperforms baselines, the outperformance is substantial. Otherwise, the method typically performs on par with the previous state of the art. Indeed, we believe our method is one of the largest jumps in performance on this task reported. Moreover, we note competitive baselines such as RBF GP and Ensembles are dramatically more expensive than our method.
While we report baselines from (Watson et al. 2021), we note several things. First, part of the reason we built upon their results is the robustness of their experimental procedure. They perform 20 trials in which the train/val/test sets are chosen randomly, and the val set is used to automatically select the termination point of training. We follow this procedure in our experiments. On the topic of normalization, we believe our normalization matches that used in (Watson et al. 2021). In particular, they say that they use a zero-mean, unit-covariance prior “in whitened data space”. It is unclear if this denotes a direct transformation of their data (which would potentially skew reported NLLs) or indicates that prior terms are transformed to match the data. We center the data to be zero-mean, and normalize features to unit covariance in the input to the network, but do not re-scale the labels. We combine our re-scaling of the data with a zero-mean prior. Thus, comparing these strategies, we believe the practical effects of our data processing are nearly identical, although the effect of normalizing feature inputs to the network may vary.
> Classification. While "Extremely strong accuracy results" are mentioned, it just performs as well or worse than competitive baselines like Laplace or Ensembles. The same for ECE, NLL, OOD detection. where "exceptionally strong performance" is claimed.
The method is somewhat simpler than baselines, but it lacks a convincing argument for why this should matter. As the authors advertise this simplicity, there should be additional results on practical runtime improvements compared to the baselines to provide some evidence for the claim that a reader should use this approach.
We have weakened our claims. While we do not strictly dominate in all metrics, the performance of our approach is highly competitive, which is especially notable given the simplicity of the approach.
Direct comparison of runtimes for each model is difficult, because some methods are post-hoc only, some require a per-epoch step, and some require a change to each training step. Thus, timing depends on other training details such as batch size. We have included runtimes for our method in comparison to standard (non-Bayesian) network evaluation, and expanded our discussion of complexity for baseline methods.