Credal sets are sets of probability distributions that are considered as candidates for an imprecisely known ground-truth distribution. In machine learning, they have recently attracted attention as an appealing formalism for uncertainty representation, in particular due to their ability to represent both the aleatoric and epistemic uncertainty in a prediction. However, the design of methods for learning credal set predictors remains a challenging problem. In this paper, we make use of conformal prediction for this purpose. More specifically, we propose a method for predicting credal sets in the classification task, given training data labeled by probability distributions. Since our method inherits the coverage guarantees of conformal prediction, our conformal credal sets are guaranteed to be valid with high probability (without any assumptions on model or distribution). We demonstrate the applicability of our method to natural language inference, a highly ambiguous natural language task where it is common to obtain multiple annotations per example.
Paper
Similar papers
Peer review
Summary
This paper proposes a method for predicting credal sets in classification tasks, generating conformal credal sets that are guaranteed to be valid with high probability without any assumptions on the model or distribution. The method is applicable to both first- and second-order predictors. Experiments on ChaosNLI and CIFAR10-H demonstrate its effectiveness and how these credal sets allow for practical uncertainty quantification in AU and EU.
Strengths
1. The paper is well-written, with smooth and clear demonstrations. Despite containing many formulas, it is easy to understand. 2. The proposed method combines imprecise probability and conformal prediction to leverage the advantages of both, which is interesting. 3. The experimental design on small label size datasets is comprehensive. 4. The limitations are discussed and soundly pointed out.
Weaknesses
1. The technical novelty is moderate: the proposed method is very similar to a previous conformal prediction method on ambiguous labels [1], as both assume reasonable probability labeling and train a standard (first-order) probability predictor (i.e., the score function). Besides, obtaining a probability interval is technically not very novel. 2. The computational complexity limits the general application of the method to large-scale datasets. 3. Access to ground-truth probability labeling seems unreasonable, especially when model calibration itself is already a significant challenge. 4. The key bounded noise assumption for the labeling process may not be very practical, as mentioned in the paper. [1] Stutz, David, et al. "Conformal prediction under ambiguous ground truth." arXiv preprint arXiv:2307.09302 (2023).
Questions
1. Would any model calibration methods provide probabilities that satisfy the bounded noise assumption? 2. While some frequentist methods also built on the first-order probability labeling, they realize it as a deterministic action at every sampling step. Would it be possible to adopt this strategy to avoid the assumption? Would the complexity be acceptable?
Rating
5
Confidence
4
Soundness
3
Presentation
4
Contribution
2
Limitations
The limitations include the implicit representation of the credal set, the intractability on large label sizes, and the assumption of access to reasonable probability labelings. The authors have adequately discussed these limitations.
Summary
Authors propose a new uncertainty estimation method based on credal sets predictors. The new approach is based on the Conformal Prediction framework and shares its finite-sample and distribution-free guarantees. Authors verify the validity of their approach on existing NLP and CV datasets.
Strengths
* theoretical part is based on an established mathematical framework and authors formally prove the validity of their method * authors provide full (anonymized) source code of their experiments
Weaknesses
* The proposed 2nd order score function seems to be not well- defined. For example, if any of $\theta^x_k$ are small than 1, then maximum in denominator of equation (8) becomes infinite and score function becomes constant. This is undesirable behavior to the best of my understanding. * While it is straightforward to assess the influence of the noise in terms of perturbations of score function, the characterization of the noise in such terms seems not natural. The most practical noise is the one that is due to the categorical observations of the label distribution. It seems that there is a gap between these two characterizations of noise. * the ability of the method to clearly separate (“disentangle”?) aleatoric and epistemic uncertainty is not fully understood. Judging by Fig.4,6,12, there is a clear functional dependency and in the main text part 4 we read: “approaches with first-order predictors lack adaptivity” with regard to a similar issue. Perhaps a special synthetic experiment can provide more insight whether it is an issue with the base method or the proposed approach.
Questions
* Can you provide a theoretical result in noisy case in terms of the standard observations, i.e. one class per data point? * Can you fix an issue with the score function? * On Figures 4,6,12 the data point collapse to a low-dimensional manifold. It is especially visible on Figure 4, where there is a clear linear dependency between AU and EU at least on a portion of the data. In light of this evidence, how can authors support the claim of part 6: “effectively capture both epistemic and aleatoric uncertainty”? * Authors introduce “uncertainty interval” as an additional uncertainty measure and report it in their figures. Part 4 also contains the following passage: “covering the entropy of a distribution with the uncertainty interval does not equate to covering the distribution itself with the credal set”. Could you please further explain the meaning and the role of this additional measure and why covering the (predicted) entropy may be desirable? * Conformal prediction can produce sets of different size depending on the non-conformity measure. Figure 3 reports the average set sizes as efficiency for the selected non-conformity measures. How does it affect the downstream task of uncertainty estimation? E.g. conformal prediction literature values smaller sets, is it useful for the UE task? * In the literature on conformal prediction, a stronger property than marginal coverage (equation 1) is of interest: conditional coverage [1]. While this might be out of scope for the current work, was this property of the proposed method investigated?
Rating
4
Confidence
4
Soundness
2
Presentation
3
Contribution
2
Limitations
I think that the majority of limitations are well discussed. However, I think that the current theoretical result being written in terms of the noise in score function is a significant limitation.
Summary
The paper shows that if access to the (even noisy) estimates of the conditional categorical label distribution is provided, one can define a confidence region within the simplex of conditional probabilities, that is guaranteed (marginally) that the true aleatoric uncertainty (ground truth label distribution) is within the predicted credal set with $1 - \beta$ probability. The authors define various score functions based on distances in the probability space and likelihood.
Strengths
The problem is really an interesting one. It is still interesting to have guaranteed uncertainty quantification and this paper is a step towards that goal. The paper is technically sound. I did not detect flaws or issues in the paper at the end. The experimental setup is complete.
Weaknesses
Please see the questions.
Questions
1. I believe that the results on the imprecise first-order data (theorem 3.2) can be enhanced leveraging the results from [1]. This study derives the conformal guarantee for the clean ground truth label, from the noisy prediction with a correction over the bias in noise (which in symmetric noise assumption it is 0.5). However, the result I mentioned concerns the prediction and adapting it to classification would be a different approach out of the context of this review. 2. Is there any way that the authors can derive a guarantee over the sampled labels as well? By that I mean deriving the conformal prediction based on labels sampled from the ground truth distribution (in cases that the non-soft label exists like CIFAR10) and an algorithm that returns prediction set from the credal sets. 3. If the answer to (Q2) is yes then what does the conformal quantile threshold look like in the simplex space? 4. Can we directly use some notion of uncertainty quantification (like log-likelihood) through the conformal risk control? I mean in Fig. 3. I see some of the credal sets are far from the ground truth distribution. Can the result enhance if you bound the TV distance in average via risk contol? 5. Are the scores that the authors used always producing symmetric convex halls? By symmetric I mean that the output is always a function of the predicted uncertainty and the computed quantile. If yes is there any way to remedy that? [1] Guille-Escuret, Charles, and Eugene Ndiaye. "From Conformal Predictions to Confidence Regions." arXiv preprint arXiv:2405.18601 (2024).
Rating
8
Confidence
4
Soundness
4
Presentation
3
Contribution
4
Limitations
I do not see any limitations that are not addressed or mentioned.
Summary
The authors propose a method for predicting credal sets in the classification task, given training data labeled by probability distributions. The resulting conformal credal sets are guaranteed to be valid with high probability (without any assumptions on model or distribution). The applicability of the method on ambiguous classification tasks for uncertainty quantification is demonstrated.
Strengths
1) The methods is a novel and promising proposal. 2) Experiments demonstrate interesting results of the proposal. 3) The paper is overall well-structured, some parts are easy to follow.
Weaknesses
1) There are no baseline comparisons. For instance, seconde-oder distribution methods like Bayesian approaches and some limitations are mentioned in the Introduction. It is unclear what are the added gains of the proposed method in terms of uncertainty estimation, compared to the existing tools like Bayesian neural networks, deep ensembles or conformal prediction framework methods. 2) Following point 1), Appendix F shows that the proposal can take up to a few hours to create the credal set at the test time compared to the training and calibration steps for a few minutes. From the practical point of view, it might be a significant drawback when the added gains are unclear. In terms of the paper presentation, it would be better that the main body at least indicates that the computational cost is discussed in the Appendix. 3) The paper seems not discuss how to predict the class index from the generated credal sets. Having a clear strategy of making class prediction would be a benefit for the end-users in practice. A recent paper [1] may provide some inspirations. 4) Uncertainty quantification is a core element in this paper. However, from the results in Figure 4, 5, and 6, it is a bit hard conclude the quality of the uncertainty estimation. Why some down-stream tasks, like Out-Of-Distribution detection or Accuracy Rejection [2], are not used for the evaluation? [1] Caprio, Michele, et al. "Credal Bayesian Deep Learning." arXiv e-prints (2023): arXiv-2302. [2] Shaker, Mohammad Hossein, and Eyke Hüllermeier. "Ensemble-based uncertainty quantification: Bayesian versus credal inference." PROCEEDINGS 31. WORKSHOP COMPUTATIONAL INTELLIGENCE. Vol. 25. 2021.
Questions
1) In lines between 107-109: Could authors please explain more how the proposed method tackle the trade off achieving the high probability of courage (tend to increase the size of credal sets) and being informative? 2) In lines between 124-127: Could authors please explain more why the Dirichlet distribution assumption is a meaningful learning? Would it also an biased estimation of uncertainty due to the lack of the ground-truth values? 4) In lines 141-142: It is stated that both predictors can also be trained using zero-order data. Could the also add some baseline compares like deep ensembles, BNNs or conformal predictions, using zeor-order data to demonstrate the outperformance of the proposal? 5) Could the authors indicate what is the potential application for the method?
Rating
3
Confidence
4
Soundness
2
Presentation
2
Contribution
2
Limitations
Some limitations are discussed in the Limitations Section. Additional limitations please refer to Weaknesses and Questions.
Response to the rebuttals
Dear Authors, Thank you very much for your engagement in the rebuttal. After carefully reading your responses, I think my main concerns remain unresolved. For better clarity, I listed them as follows: **No baseline comparison with other existing uncertainty representation methods. The added gain of the proposal remains highly unclear.** As you indicated in Eq. (4) in the paper, we could train a standard neural network whose prediction is a single probability distribution from the first-order data with probabilistic ground truth by minimizing the cross entropy (Kullback-Leibler divergence). Therefore, deep ensembles can be built in this context. Similarly, I believe that Bayesian neural networks (BNNs) can also be trained using first-order data. Therefore, *'While the methods mentioned in the review rely on zero-order data, our approach requires first-order data'* in your response is not very convincing to me, because we could the existing baseline methods do not have limitations in using first-order data. As your proposed method requires at least first-order data for calibration, the comparison could be conducted in two aspects to enhance the contribution and soundness of your work potentially: a. Train some existing second-order methods (e.g., deep ensembles, BNNs, etc.) on zero-order data and credal predictor on zero-order data with first-order data for calibration. b. Train all the baselines on the first-order data. I acknowledge that credal sets are different representations of uncertainty compared to deep ensembles, BNNs, evidence-based deep learning, etc. However, some downstream tasks are widely used in the community to compare different uncertainty representations. For example, in the evidential deep learning paper [A], where Dirichlet distributions are used for uncertainty representation, they compared their methods with other representation methods, such as the BNN model and deep ensembles for performance comparison. As for credal representations, for instance, the paper [B] also compared credal inference with Bayesian approaches. The paper [C] also compared their method, a credal self-learning approach, with some strong baseline to convince the audience of the added gain of their proposal. [A]M. Sensoy et al., "Evidential deep learning to quantify classification uncertainty." [B] Shaker, Mohammad Hossein, and Eyke Hüllermeier. "Ensemble-based uncertainty quantification: Bayesian versus credal inference." PROCEEDINGS 31. WORKSHOP COMPUTATIONAL INTELLIGENCE. Vol. 25. 2021. [C] Lienen, Julian, and Eyke Hüllermeier. "Credal self-supervised learning." Advances in Neural Information Processing Systems 34 (2021): 14370-14382. *Therefore, I believe adding some baseline comparisons would strengthen the contribution of this work.* **Complexity discussion** Thanks for the clarity on the complexity discussion. 11 seconds per test point seems acceptable. I think by adding a cost reference (for instance a baseline), the audience (end-users) can better sense the added relative complexity overhead. *'For this dataset, the calibration step and uncertainty quantification were completed in less than a second.'* Does it include the time cost of solving the optimization problem in Eq. (12)? **Deriving the class index (set) from the generated credal set as well as the potential applications** I thank the authors for elaborating on the possible ways of doing that. From a practical point of view, I believe it is necessary to have such a strategy. Compared to, for example, the Bayesian approach delivering a class index as the final prediction, conformal prediction generating class index sets, only have a credal set as the end can induce high complexity to derive a class prediction as well as interpret and apply the results to the downstream, e.g., decision problems. As the proposed method requires at least first-order data for calibration, could the author enlighten me on the potential application (e.g., the first-order data widely exists and is used for classification) of this method? I do think that the work is interesting and has potential. However, I think that it needs more work before publication.
Thank you for engaging with our rebuttal. We're glad you find the work interesting and see its potential. We'd like to provide further details on the points you mentioned. - **Comparison to baselines**: We thought again about your remarks and would summarize our point of view as follows: - Credal sets and second-order distributions are different uncertainty representations, and you also agree that a direct comparison is not possible -- it's like comparing apples with oranges. The same applies to uncertainty measures applied to these representations, which are of a different nature and not commensurable. - Since a credal set cannot be turned into a second-order distribution, but a second-order distribution can be turned into a credal set in a more or less natural way via thresholding (like deriving a confidence interval from a density function), the only meaningful comparison we can envision is in terms of credal sets produced by different methods. What we could imagine, therefore, is a comparison of the credal sets produced by our method with the credal sets obtained, e.g., as the convex combination of the predictions of an ensemble of deep neural networks (trained on either zero- or first-order data), similar to what Shaker et al. did in reference [B] that you provided. The comparison could be done, e.g., in terms of the precision/efficiency and the coverage/validity of the credal sets, i.e., in a Pareto-sense. Before starting such experiments, it would be important to know whether that would be in your favor. - Comparing the performance of different methods on downstream tasks is potentially also possible. However, we intentionally avoid such comparisons, which have their own problems and can compare uncertainty quantifications only indirectly -- in the literature, they are only used in the absence of first-order data (e.g., through measures such as coverage, efficiency, or other metrics presented in Figures 4 and 6 or the new measure proposed in our main rebuttal response). However, given access to first-order data, we can evaluate performance in a more direct way, which is clearly preferable. Also, shifting the focus to downstream performance at this point would change the scope of the paper substantially. - **Complexity discussion**: We're happy that our responses addressed your concern and appreciate your suggestion. Regarding your question, yes, finding the distributions with maximum and minimum entropy was included. Please note that we are using simplex discretization, which makes these calculations extremely fast and efficient. - **Deriving the class index (set)**: We're glad our responses were helpful. Regarding your question, as we mentioned in the paper, datasets in which multiple human experts annotate each data instance are becoming increasingly available in practice, including widely used benchmarks like ImageNet [1] and CIFAR-10 [2]. Crowdsourcing frameworks further facilitate such annotation mechanisms, making them even more accessible. (Please refer to our extended related work in Appendix A for additional references.) One particularly interesting and important case is in the medical domain, especially in dermatology [3], where multiple doctors evaluate the same image to classify a skin disease, often with conflicting opinions. This conflict can be aggregated into a probability distribution over labels, effectively representing aleatoric uncertainty, which is often overlooked. [1] L. Beyer et al., "Are we done with imagenet?" [2] J. C. Peterson et al., "Human uncertainty makes classification more robust" [3] D. Stutz et al., "Conformal prediction under ambiguous ground truth."
Dear Reviewer ieV3, Regarding W4, we would also like to draw your attention to our response to W2 of Reviewer LLVn, where we explained how the bounded noise assumption translates into the noise arising from estimating the ground truth first-order distributions using only samples from that distribution. Also, given that we have addressed all your questions and concerns, we'd love to know if there are any additional ways to improve our work so that it becomes more convincing and impactful, beyond borderline acceptance, for both you and the broader community in the future. Best regards, The Authors
I have carefully read the authors' response and appreciate their efforts in answering my questions. However, I maintain my opinion that the contribution and impact are moderate. Although there are synthetic datasets with probabilistic labeling, I believe the most important application of inherent labeling uncertainty is in the medical domain, which the authors did not address in the experiments. Given that current medical machine learning models often suffer from poor performance, may I ask how this would influence the conformal credal sets?
Thank you for the feedback. We would like to highlight that our main experiments and results, presented in the main part of the paper, are conducted on two **real** datasets: **ChaosNLI** and **CIFAR10-H**. We found ChaosNLI particularly intriguing due to the notable disagreement among annotators, indicating high aleatoric uncertainty. Additionally, the feasibility of visualizing the credal sets within a 2-D simplex adds to its appeal. We included CIFAR10-H in our experiments because CIFAR10 is widely used in numerous studies, and we aimed to demonstrate the applicability of our method on such datasets to gain a better understanding of uncertainty. In the medical domain, we are only aware of the Google Dermatology DDX dataset [1], where the task is to classify a skin image into a specific skin disease. However, For this dataset, only the annotations and predictions of a model are available, and unfortunately, the images are not publicly accessible (please see [here](https://github.com/google-deepmind/uncertain_ground_truth)). This limited us to conformalizing that given model, which did not provide sufficient flexibility for our experiments. Nevertheless, we can incorporate this dataset into the camera-ready version of the paper if you find it beneficial. Regarding synthetic data, we employed it for two specific purposes: first, as an illustrative example to show the behavior of our credal sets constructed with different models as the number of training data increases and epistemic uncertainty decreases to (almost) zero. Second, we used synthetic data to evaluate the performance of our algorithm under the assumption of imprecise first-order data, which is impossible to test with any real dataset. Therefore, we believe that these synthetic data experiments were necessary to further demonstrate the potential of our work. Regarding the raised question, poorer model performance translates to a greater distance between the true first-order distribution and its prediction for the first-order predictor or to a lower predicted density for the true first-order distribution given the second-order predictor. In either case, this generally results in higher nonconformity scores, which eventually lead to larger credal sets. These larger credal sets represent higher epistemic uncertainty, indicating a greater lack of knowledge. [1] D. Stutz et al., "Conformal prediction under ambiguous ground truth."
Dear Reviewer LLVn, Given that we have addressed all your questions and the raised weaknesses, we are wondering if there are any remaining points that are unclear or require further explanation for us to clarify and improve our work. Best regards, The Authors
Rebuttal well received
Dear authors, your rebuttal was well received and you partially addressed my concerns. I will decide on the score changes after the discussion with other reviewers.
I thank the authors for their reply. After reading the authors response I believe that my questions are answered in a convincing way. Still, I will not change the score as it is already high. Here I break down my idea about the replies. 1. For Q1 I totally mentioned a new possible direction and therefore I do not expect the authors to modify the current state of the paper based on that. Their comment was clear and enough. 2. for Q2 and Q3: I believe the illustrative solutions work with a guarantee higher than 1 - alpha and not concentrated around it since the approach will be conservative. Intuitively I think that one simple approach is via sampling from the credal set and then defining the prediction set. But this problem is really confusing and addressing it is outside of the scope of this review/comment box. 3. I still think that one additional (and easy to experiment) study could be an ablation over different risk functions. But even without that I think the current state of the paper is acceptable. **Why not increase:** I think that the current score is already high enough and I do not think that the paper is an award-winning paper. **Why not decrease:** Despite the difference in my score to other reviewers, I still do not decrease my score; here I can address some key points from reviewers with lower scores and why I do not find their comments a reason to reject the paper: 1. **No comparison with other uncertainty quantification papers:** Other methods like deep-ensembles, Bayesian deep learning and dropout, provide uncertainty quantification without any notation of guarantee. Their output is a probability on the top predicted class and not a joint quantification of confidence over all classes. I think in general, comparing Bayesian UQ approaches with a frequentist method like conformal prediction is slightly off topic. Vanilla CP concerns ranking and therefore there is no priority among any two labels that are in the prediction set. Similarly, Conformal Credal Sets also should not prioritize any point inside the guaranteed credal set at first place so other UQ methods are not (at least trivially) comparable. 2. **Question on how to use credal sets:** It is pointed out by some reviewers that the credal sets (2nd order probability space) does not clearly show how to use in end-point prediction task. Although the method is a bit far from a good end-user experience, a new direction of work could be to interpret their results into meaningful probability alongside the top label etc. Please note that the credal sets can be defined independent of the architecture of the model and training procedure. I believe leaving these interpretations aside, the question of how to use a frequentist approach to decide a probability over labels is still a strong question to answer. After all I also believe that some terminologies (like noisy labels) are a bit unclear and I strongly encourage the reviewers to rethink the writing and some terms used in the paper. This was also an encouragement (supported clearly by other reviews) to decrease the score since the paper does not deliver a clear intuition to someone who reads it without any background on basics of CP and its differences to non-frequentist approaches of uncertainty quantification.
Decision
Accept (poster)