Pearls from Pebbles: Improved Confidence Functions for Auto-labeling

Summary

This paper introduces a threshold-based auto-labeling (TBAL) method called Colander to maximize TBAL performance by finding the optimal labling confidence function and thresholds. In order to find the optimal confidence function, Colander treats the auto-labeling objective as an optimization problem that maximizes the coverage under label error constraint. They use a neural network as the confidence function, and design a surgate for the optimization problem which can then be optimized using gradient-based methods.

Strengths

S1: The paper turns auto-labeling into an optimization problem, and proposes a disciplined solution that can be solved by gradient methods. S2: The proposed optimization surrogate may be adopted by other auto-labeling methods. S3: The paper presents extensive experiments to compare with existing methods, and the results are promising.

Weaknesses

W1: the paper is overall well-written, but the discussion of the thresholds is confusing or missing details. Specifically: - In line 89, it says "the vector _**t**_ denotes scores over _k_ classes", and I suppose score means the predicted scores (or probabilities) of the _k_ class, but later _**t**_ is defined as a vector of thresholds. - In line 144 (P1), it is unclear what _T^k_ stands for, and how the set of threshold, _T_, is determined. - In Algorithm 1, the Colander produces the estimated confidence function and thresholds hat _ti'_ (line 14), but the threshold is not used, and it relies on Algorithm 2 to estimate the threshold for each class. There is no discussion what the difference between these two thresholds. W2: There is no discussion of the computation overhead of Colander.

Questions

Q1: How do you determine the threshold set of _T_? Is it a grid of _k_ classes * _n_ steps in [0,1]? Q2: Why not use the thresholds produced by Colander (line 14 of Algorithm 1)? Q3: My understanding is the that confidence function is searched by a neural network using gradient descent, and the thresholds are searched by iterating the training process over the threshold set T. But can we also make the thresholds a neural network (meaning that thresholds would be sample-specific)?

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

The computation cost of Colander increases linearly with the size of the set of confidence thresholds _T_, but it seems the process requires finer granularity in _T_ in order to get a better estimate of the optimal confidence function and thresholds.

Summary

This paper proposes a novel auto-labeling method, called Colander. In contrast to existing works, Colander models the objective of finding an optimal confidence function as a constrained optimization problem (the confidence function should have maximum coverage whilst obtaining a sufficiently low error rate; both components are controlled with a penalty term). Colander is evaluated and compared to baselines on four datasets covering vision and language tasks: MNIST, CIFAR-10, Tiny-ImageNet and 20 Newsgroups. The obtained results clearly indicate that the method improves upon existing baselines, both in terms of coverage and error rate.

Strengths

* The paper proposes a novel approach to identifying confidence functions for auto-labeling. As indicated in the paper, finding confidence functions for auto-labeling is challenging, and this paper tackles this problem elegantly via constrained optimization. * The paper’s results are promising, showing that the introduced method improves upon existing baselines. * Overall, I believe that this paper provides a solid contribution to the research area of auto-labeling.

Weaknesses

* The paper focuses heavily on formally introducing Colander, and the experimental content is comparatively thin. To provide the reader with a better notion of robustness, it would for example have been useful to provide additional details on the hyperparameter search in the main paper, and how selecting those differently affects both the coverage and error rate. Likewise, it would have been useful to provide additional details on the impact of the introduced penalty term. * It would also have been interesting to better understand the relationship between chosen model architecture for a dataset and auto-labeling performance. * The paper scarcely discusses limitations in the conclusion section, yet I would have expected a more detailed discussion of where and how this approach is limited. * Related to that, the paper does not touch upon future work / research questions that the obtained results create.

Questions

* Did you conduct additional experiments evaluating how different architectures affect auto-labeling performances for different datasets (i.e., how does changing model architectures for each of the four analyzed datasets affect auto-labeling performance)?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

As discussed in the Weaknesses, the paper mentions a limitation in the conclusion section but I'd encourage the authors to further elaborate on how and where their approach is limited, and how such limitations can potentially be addressed in future work.

Summary

This paper discusses threshold-based auto-labeling functions aimed at identifying a large subset of unlabeled instances where the auto-labeling error remains below a specified threshold. The authors observed that standard temperature-scaled softmax scores are inadequate for effectively thresholding labeled instances. To address this, they propose learning a confidence scoring function and a threshold based on a subset of the validation data. This confidence function is a two-layer neural network trained on the logits from the last two layers of the base model. Extensive experiments were conducted on both image and text datasets using various training-time strategies to optimize the base model, demonstrating the efficacy of their approach.

Strengths

1. The paper is well written, with a variety of experiments conducted to demonstrate the applicability of their proposed method. The choices of strategies and hyperparameters are documented well. 2. The performance of the proposed method is quite significant, achieving lower error rate and higher coverage than the other methods, especially on the harder datasets.

Weaknesses

1. The whole TBAL procedure is a hybrid mixture of iterative training/self training and active learning. While the authors explained the differences between TBAL and self-training + active learning (ST+AL), the difference seems small (mostly, TBAL aims to identify a low-error auto-labelled dataset and ST+AL aims for a good classifier). These two goals do not seem fundamentally different, and can be easily translated to each other. Given the similarity of the framework, I feel it is still necessary to include some ST/AL works in the experiments. 2. A lesser weakness is that due to iterative training, the proposed method and compared methods all went through rounds of data selection, which makes the comparison indirect. For example, it would be hard to tell the exact improvement on thresholding quality the method brings.

Questions

n/a

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

2

Limitations

The authors acknowledge that the limitation of their work is the requirement of validation data.

Summary

This paper addresses the challenges of overconfident scores in threshold-based auto-labeling (TBAL) systems. It critiques existing confidence scoring and calibration methods and introduces Colander, a new post-hoc method tailored to optimize TBAL performance.

Strengths

- The paper has a good identification of the problem which is over confidence of existing TBAL confidence functions, and it proposes a novel framework to find the optimal confidence function. - The paper conducts extensive experiments on both image and text data. - The paper compares various post-hoc functions. - The paper is well written with great details in Appendix

Weaknesses

- The authors rightly mention limitations (“A limitation of Colander is that similar to other post-hoc methods it also requires validation data to learn the confidence function. Reducing (eliminating) this dependence on validation data could be an interesting future work.”) in the conclusion. So, does this mean that they use Gold data for validation? If that is the case, that might be the weakness of TBAL in general. The following paper talks about it in detail: https://aclanthology.org/2023.acl-long.796/

Questions

- Not a question but authors can consider following related paper for comparison. It also works with confidence scoring: https://aclanthology.org/2021.naacl-main.84.pdf

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

Yes.

Summary

This paper introduces a Colander for threshold-based auto-labeling (TBAL) pipeline. Colander trains a flexible confidence function using latent information of the classifier model, instead of fixed confidence functions. The optimization problem of Colander is based on the objective of TBAL, that is, maximizing coverage while minimizing auto-labeling errors. The authors consider differentiable surrogates for the 0-1 variables using the sigmoid function to make this problem tractable using gradient-based methods.

Strengths

This paper is well-motivated. The proposed method is explained with simplicity and clarity.

Weaknesses

Contribution: Although this paper proposes the Colander for the TBAL pipeline, there is no theoretical analysis and understanding of mathematical properties. Completeness: - The paper includes unknown notation, which is used without pre-defined. - In Section 3.3, an undefined procedure exists: RandomQuery, RandomSplit, and ActiveQuery. In the case of Random method, it is possible to guess by readers, but in the case of ActiveQuery, a detailed explanation is needed. - There are a few expressions that make it difficult to figure out the intent. Additionally, some keywords used in the paper need to be unified. - What is the difference between the score (or scoring) function and the confidence function? - What is the difference between ‘inherent tension’ (in line 33) and trade-off? - What is the difference between post-hoc method and calibration method? - The reviewer is unsure if Figure 1 is necessary for the reader: it contains acronyms that are not defined in advance, such as ERM, and certain formulas (h) are left undefined in the introduction. - Line 174, line 186: The reader does not know which Appendix to look at because Appendix information is omitted. - Appendix, Algorithm 2, line 8: The formula where C_1 weight is considered is not defined in the text. - Appendix, Table 3: there are typos. - This paper is considered to have no significant contributions compared to previous research about TBAL[1]. [1] Vishwakarma, Harit, et al. "Promises and pitfalls of threshold-based auto-labeling," Neural Information Processing Systems, 2023.

Questions

Active Learning Vs. TBAL - Active learning and TBAL fundamentally play the role of labelers for exploiting unlabeled raw data. Lines 100-104 include these differences, but it is hard to get the point. Please explain these differences in detail. - Although the paper includes a performance comparison with active learning (and self-training), an algorithmic setup of active learning is not provided. Please provide experimental settings for understanding whether it is fair or not. - Why does the TBAL process require a human labeler at every iteration? Is it impossible to use labeled and unlabeled data completely separately in advance? Colander (proposed solution) - In Algorithm 1 (line 14), Colander extracts a confidence function g ̂_i and t ̂_i^' through an optimization problem. As the reviewer’s knowledge, t ̂_i^' is different from t ̂_i obtained estimation threshold process. Where is t ̂_i^' used? - The optimization problem (P1,2,3) is solved using a given classifier. It means that this optimization process only works when the classifier's performance is guaranteed. Is it right and is this a correct assumption? - The reviewer understood that the proposed solution configures the confidence function as a neural network, so it is flexible. But what is the meaning of 'choice of confidence function.' Does Colander learn multiple confidence functions and choose one of them? - Why did the authors select exactly two layers of the classifier as inputs to the confidence function? There are various options: single layer, three layers, and inclusion of output information. - Is it right to learn a new classifier/confidence function every round? If so, how long does the entire process require? If not, some expressions in the algorithm (pseudocode) and Section 3.3 need to be changed. Post-hoc method - Most baseline methods were proposed before 2020, except one algorithm. Please check following methods. - R. Verma, et al. "Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles." AISTATS, 2023. - L. Tao, et al. "Calibrating a deep neural network with its predecessors." IJCAI, 2023. - T. Joy, et al. "Sample-dependent adaptive temperature scaling for improved calibration." AAAI, 2023. - The reviewer would appreciate it if the authors could provide some insight/intuition into the problems with previous methods. The experimental sections just show empirical results and lack discussion. Empirical results - It is important to provide accurate settings when performing partial improvements in the overall pipeline. Have you checked the performance of various procedure method changes such as ActiveQuery? - Are there any performance analysis results regarding changes in confidence scores? - The experimental results clearly show that the algorithm has high performance in the provided experimental settings. What about time efficiency? Have you checked the wall-time? - In the optimization problem (P3), how did the authors select the upper-bound error parameter ϵ? How do experimental results change depending on parameter changes? - The value of C_1 in line 8 of Algorithm 2 seems to be fixed at 0.25. What results can see when it changes?

Rating

4

Confidence

3

Soundness

3

Presentation

1

Contribution

2

Limitations

In this paper, there was not enough discussion about the limitations.

Thank You!

Thank you for your response. I enjoyed reading your paper and learned a lot. A few things regarding the reviewer YvdP. As I differ a lot in scores, I felt I should comment on a few of them. - The presentation of the paper is really good. I do believe that Figure 1 was useful while I was reading the paper and two of my colleagues also agree with that. - Many comments are unreasonable for example "There is no theoretical analysis and understanding of mathematical properties" Good luck with the other reviewers!

Thanks for the detailed response!

Thank you for providing such a detailed response and the additional results and insights. I will maintain my score and recommend acceptance as indicated in my initial review.

Please be polite during the discussion

Dear reviewers, Thanks for responding to the authors' rebuttals! Please try and be polite during discussions. Everyone has a right to have their own opinions on the submission, and nobody should be pressured to change the review or rating. Thanks!