Gaussian Process Probes (GPP) for Uncertainty-Aware Probing

Summary

This paper provides Gaussian process probes (GGP), a probabilistic method to evaluate uncertainty for a binary classification task over a (pre-trained) feature extractor. The core idea is to use GP instead of a linear probe on the feature extractor. The use of GP provides a natural way to estimate two uncertainty measures: aleatory and epistemic. To examine the performance of GPP, the authors conduct two experiments using synthetic and real images, showing that GPP can preferably estimate uncertainty.

Strengths

1. Novel formulation to evaluate uncertainty for linear probe problems. 2. The paper is well-written and easy to follow. A lot of illustrative figures help to understand the key concepts. 3. Solid derivation of the GP model and uncertainty estimation.

Weaknesses

1. The experiment results (Fig 5) look somewhat strange. For me, the LPE result doesn't make sense. The setting here is that only positive labels are noisy (flipped to negative with 50% probability), and the negative labels do not contain noise. In such a situation, the well-trained classifier will output 0.5 as the judged probability for positive and 0 for negative examples. This should occur when the classification problem is linearly separable on the feature space, and the number of observations is large enough. I assume both conditions are satisfied when the number of observations is 128, but I cannot see such a tendency from LPE. Also, due to the imbalanced noise, I expect to see a vertically asymmetric behavior for LPE (e.g. the upper limit of judged probability would be close to 0.5), but it is not. 2. This paper doesn't explicitly explain multiclass cases. Line 111 says extending GPP for multiclass problems is straightforward, but I don't feel it is trivial. Also there are no experiments for multiclass cases. 3. Some technical descriptions are unclear. See "Questions" below.

Questions

1. In 154, it is said the kernel k is defined as "k(a, a) = v", but the definition (3) tells us that "k(a, a) = v (||a||^2 + 1) / (||a|| + 1)^2", which is not v unless a = 0. Why does this inconsistency happen? 2. Line 144 says that GP with the kernel (3) is "equivalent to defining a distribution over linear latent functions", but then why does Fig 1 show some nonlinearity? 3. What is the training procedure of LPE in Fig 5? Did you confirm that LPE was properly trained?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors fairly address the limitations.

Summary

This work introduces a unified framework called Gaussian process probes (GPP) for probing and quantifying uncertainty in models' representations of concepts. GPP extends linear probing methods and uses a Bayesian approach to estimate the distribution of classifiers induced by the model. This distribution measures the model's ability to represent concepts and provides a measure of confidence in the representation. GPP is a simple procedure applicable to any pre-trained models with vector representations, requiring no access to training data, gradients, or architecture details. The validation experiments on synthetic and real-world datasets demonstrate that GPP can effectively probe concept representations with a small number of examples, accurately measure both epistemic (confidence) and aleatory (fuzziness) uncertainties, and detect out-of-distribution data.

Strengths

The paper addresses an important problem of understanding the inner representations of complex models. It does so by providing some estimates over two different types of uncertainty. It gives clear explanations for the epistemic and aleatoric uncertainties. The use of GPs and the study of two types of uncertainty in this particular context seems novel.

Weaknesses

As a non-expert in the area of probing, I found some parts of the paper hard to follow (see Questions below).

Questions

Can you explain in more detail the choice of prior as outlined in Section 2.3.1. I'm not sure I understand what you mean by matching the Beta prior and matching the normal distribution in lines 125-126. I think this must impose important assumptions on the model, affecting the outcomes so it would be good to communicate it clearly. Are there any practical implications to considering the two types of uncertainty separately? Are there any cases where it might give counterintuitive results? Do you foresee any practical issues with using higher dimensional Dirichlet-based GPs? Can the inference still be performed in closed form?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors outlined limitations in the main part of the paper and provided a broader impact statement. From a practitioner's point of view, it would be important to outline the exact assumptions that the GP model imposes (I assume those will depend on the specific dataset/model). I believe this is done to some extent in the paper but I have further questions (as mentioned above). The quality of the writing could be improved, it requires a proof-read. Minor: Line 69 - best repeated Line 124 - uniformly distributed

Summary

The manuscript proposes a Gaussian process-based probing (monitoring a layer using only the layer’s activations without influencing the model itself) method that can estimate uncertainty in prediction. In the experimental result section, the proposed method is applied to several example datasets that can check whether the proposed method can correctly estimate fuzziness in concepts and to out of distribution (OOD) detection.

Strengths

I think that the text is easy to read and understand. This work can be significant because we need more tools to understand deep neural networks, which are black-box models. Uncertainty quantification is also a major selling point of the proposed method. However, I hope that the manuscript would have provided monitoring and diagnostic tools to understand layers’ behaviors using estimated uncertainty.

Weaknesses

1. I think that the novelty of the proposed method is incremental. In fact, a number of works have been proposed based on GP or Deep GP models for uncertainty estimation (including decomposing uncertainty into Aleatoric and Epistemic uncertainty) and out-of-distribution (OOD) detection. The authors put together several ideas (from existing work) to create a new method that can be a good tool for certain problems. However, I did not find any new innovations or improvements from the methodologies of the proposed method. 2. I think that the manuscript could be further improved in terms of presentation. My main concern is that the motivation for the use of Beta (Dirichlet) Gaussian processes is not clearly stated in the main text. Why not just use the original Gaussian process formulation for classification? This approach requires approximation for inference, but there already have been accurate and practical tools, e.g., expectation and propagation (EP). If the main motivation is about computational complexities (as in the reference paper for Beta GP [ref Milios et al., 2018]), then the examples included in the experiments do not seem to be well designed to get the readers appreciate this motivation from them (as the numbers of the training datasets appear to be small).

Questions

In eq. (4), what is \mu(a)? I could not find its definition in the text. Regarding eq. (4) (and eq. (2)), it appears that we don’t need to consider two linear functions, W_alpha and W_beta, because they can be reduced a single linear function f = f_alpha - f_beta = W^T \psi(x), where W=W_alpha - W_beta. What did I miss here?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

Yes. No potential negative societal impact of their work.

Summary

- The authors propose a probabilistic probing method to understand a given pre-trained classifier. - The authors describe how looking at classifier predicted class probabilities is not enough since "0.5" in a binary task can happen for several reasons spanning the aleatoric/epistemic uncertainty spectrum - On the other hand, the proposed probabilistic method allows for posterior estimates of classifier's predicted probabilities, which allows for example to produce variance/entropy/etc - The particular method used is to compute functions g() of a classifier's representation a(x) for inputs x and to study the distribution of g() - The distribution of g() is defined through a hierarchical GP called the Beta GP - After giving an exposition of the model the authors introduce two sets of experiments: whether the probe can correctly identify that there is true label uncertainty in some synthetic experiments, and whether the model can correctly identify that data is OOD for a given classifier. While proposed method is well motivated and properly defined, I have some major questions regarding evaluation (proper definition of all tasks and baselines). For now, I think the paper needs some clarification before acceptance, but would be glad to raise my score, given clarification from authors.

Strengths

The papers strength's are: - Beta-GP-based probing method is very clearly defined - All of the computations e.g. posterior computations are precisely stated in the appendix - The overall motivation of the work (probabilistic probing) is solid - The introduced entropy/variance based metrics that pull apart some aspects of aleatoric versus epistemic uncertainty, adapted from previous literature, seem like a nice contribution to uncertainty/probing evaluation metrics

Weaknesses

There are two downstream uses of the method - checking correlation of probes' reported uncertainty versus the true label uncertainty for synthetic/semi-synthetic datasets - OOD detection The precise definitions for baselines in uncertainty estimation experiments were not given (namely LPE) and more definitions are necessary to correctly interpret the variance/entropy-based metrics. See "Questions". For OOD detection, the superiority over baselines is exemplified but the baselines are out of date. The synthetic dataset is not named, and some details are missing from the Imagenet result to understand exactly how to the pretrained model checkpoints were run on the binarized data. See "questions" Minor writing style suggestion that does not affect my review: In sentences like "There are important details in Beta GPs that require special attention: how to set the prior and how to approximate the posterior." Since there are a few distributions floating around that were recently introduced to the reader, it could be helpful to include the symbols in mid-sentence: "There are two important details in Beta GPs that requires special attention: how to set the prior (mu and k(,) in GP(mu,k)) and how to approximate the posterior p(f_alpha,f_beta | D, mu k )." And include a sentence after "With samples from the posterior (falpha, fbeta) we can then approximate the distribution of g()~G with the log normals parameterized by f_alpha, f_beta." This could help the reader piece everything together.

Questions

The results for the first experimental section (uncertainty quantification) seem convincing relative to the baselines but one major issue - LPE is not cited, and the authors do not mention that they propose it. There is only a high-level description given. I was trying to look for a precise definition of the method since it is taken as one of the main baselines, but cannot find it anywhere. - Please give a citation for LPE, or precisely define it and state that this is a baseline proposed by the authors. - This is moreover important since one cannot arbitrarily compare model entropies/variances with each other for continuous models without more assumptions (same base measure, etc). And for this, precise definition of all distributions is important. For OOD detection, two main groups of concerns, task definition and baselines: Task issue 1: - there seem two be two datasets, a synthetic one and ImageNet - this section doesn't explicitly name which synthetic dataset. I only assume by continuity that it is the shapes dataset from the previous subsection. Please name all datasets in all sections/captions/figures they are used in rather than just refer to "synthetic data". - The authors say "we generate another set of queries that include 1024 ID images and 1024 OOD images (uniform random noise)" but don't say from where is the in-distribution. Task issue 2: - For Imagenet there are also some missing details that stop me from understanding the experiment. - I understand this: "Queries and observations are sampled disjointly from the validation split of the ImageNet dataset" - I understand this + I read the appendix section: "We make 10 set of Ds using 10 binary classification tasks defined by supersets of ImageNet classes" - How are the pre-trained classifiers use? Did you specifically pull checkpoints that were also trained on the binarized problems? Or did you somehow pool together a multiclass classifier's probabilities for the underlying model + renormalize? What happened to the probabilities assigned to the other classes? - Please let us know and then revise the paper to clarify the exact use of the pre-trained models versus the data and what exactly was computed, including any equations if there were any transformation steps from multiclass checkpoints to the binary task, or clarify that binary classification model checkpoints were used that correspond to the same binarization that you applied to the data - Also, is the out-distribution in both datasets just uniform noise, or uniform noise added to ID data? Baselines: The baselines seem a little out of date, with the two non-LPE baselines (again, where is LPE from?) being from 2016 and 2018. It's okay to include older baselines as part of a broader evaluation, but why not include also recent methods from 2020-2023 such as any of the below. I might be missing context on this sub-area of ML, but at least in others I review actively, it's fairly rare to find a paper that only evaluates to methods from <=2018. Several well-cited recent methods are documented below. Maybe not all are applicable as baselines for your particular setup/model assumptions/method assumptions (e.g. black box vs having access to something), but please clarify. - 2018, ODIN, Enhancing the reliability of out-of-distribution image detection in neural networks. https://arxiv.org/abs/1706.02690 - 2018, A simple unified framework for detecting out-of-distribution samples and adversarial attacks. https://papers.nips.cc/paper_files/paper/2018/hash/abdeb6f575ac5c6676b747bca8d09cc2-Abstract.html - 2019 Likelihood Ratios for Out-of-Distribution Detection https://proceedings.neurips.cc/paper_files/paper/2019/file/1e79596878b2320cac26dd792a6c51c9-Paper.pdf - 2020, Energy-based out-of-distribution detection. https://proceedings.neurips.cc/paper/2020/hash/f5496252609c43eb8a3d147ab9b9c006-Abstract.html - 2021, REACT, React: Out-of-distribution detection with rectified activations. https://proceedings.neurips.cc/paper/2021/hash/01894d6f048493d2cacde3c579c315a3-Abstract.html - 2022, Dice: Leveraging sparsification for out-of-distribution detection. In ECCV, https://arxiv.org/abs/2111.09805 - 2022, Out-of-distribution detection with deep nearest neighbors. https://arxiv.org/abs/2204.06507 - 2022, Scaling out-of-distribution detection for real-world settings, https://arxiv.org/abs/1911.11132 - 2022, Vim: Out-of-distribution with virtual-logit matching, https://arxiv.org/abs/2203.10807 - 2022, a review: https://arxiv.org/pdf/2110.11334.pdf In short, it seems like good work, but it seems like authors know more than the readers (about task definitions, datasets, why newer baselines were not used). Please clarify. Glad to increase score if sufficient answers given to these concerns. Most concerns can be answered without experiments except the "old baselines" concern, which requires either experiments or a commitment to include newer baselines.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

Yes.

Summary

The authors introduce Gaussian process probes, a probabilistic probing method. They use this method to obtain additional insights into the internals of deep learning models, using the concepts of aleatoric and epistemic uncertainty.

Strengths

* This is an interesting and useful addition to the literature on probing methods. It enables obtaining insights into the function of a deep learning model that could not have been obtained before. * The method is novel to the best of my knowledge. * The method is well-presented. The methods section is well-structured and clear.

Weaknesses

* Other parts of the paper could use from better writing. In particular, I thought the intro and the experiments section could use additional polishing. * The paper could benefit from more extensive experiments. At the very least, it would be standard to test the method on more than one dataset.

Questions

* What other methods and datasets could serve as benchmarks and how would the method perform there

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

No concerns about addressing limitations

Raise my score

Thank you to the authors for the response. The explanations and the additional experiments about LPE make sense. My major concerns are resolved, and I will raise my score.