Weak Supervision Performance Evaluation via Partial Identification

Summary

This paper proposes solutions via convex programs to estimate Frechet bounds for Programmatic Weak Supervision (PWS). This approach uses estimates of the true labels via labelmodels (i.e., different aggregation schemes that exist in the literature). With these estimates of the labels, they provide an approach to estimate bounds on the accuracy (and other quantities) of the weak labelers. They provide experiments to check the validity of their bounds and also provide experiments with weak labelers generated via prompting to examine how their bounds perform under instances of weak labelers with different qualities/accuracies.

Strengths

The strengths of this paper are that it analysis the PWS setup from a theoretical perspective, tackling the problem of analysis the performance of individual weak labelers. 1. The authors provide a nice analysis of their estimation scheme. 2. They derive the asymptotic distributions of their estimators and show they are consistent. 3. They also provide CI for their estimates given finite sample data. 4. A novel application of Frechet bounds (and the corresponding literature) to the field of programmatic weak supervision.

Weaknesses

My only substantiative concern with this paper is that its applicability to the task of label set selection is weirdly motivated. Given a correctly specified label model and a better than random weak labeler (which is exactly the setting that where this method achieves valid bounds), then there is no need to prune away worse-performing weak labelers as they are helpful for the labelmodel. I agree that removing these low-accuracy weak labelers can be helpful given misspecification or a violation of assumptions, but in the setting where this paper achieves valid bounds, then I believe there is no need for this pruning.

Questions

1. Do you have any intuitions about settings where the usage of your method for label set selection is well-motivated? 2. There's also a line of work in PWS that looks to train end models directly [1, 2, 3]. Can the bounds of this work be applied directly to these models to analyze their error rates? This added discussion would be interesting to broaden the applicability of this theoretical framework. [1] Sam, et. al. Losses over Labels: Weakly Supervised Learning via Direct Loss Construction. [2] Cachay, et. al. End-to-End Weak Supervision. [3] Yu, et. al. Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach.

Rating

6

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

Yes, limitations have been adequately addressed.

Summary

The authors propose a method for bounding the performance of models obtained using weak-supervision, without the need for a gold-standard label set. The proposed approach starts with Frechet bounds, and casts them as dual optimization problems. From there, the dual optimization problems are relaxed to a smooth functions which can be solved using L-BFGS or similar methods. Finally, the authors explore the performance and utility of the proposed method on several empirical datasets. The evaluation demonstrates a non-trivial ability to bound performance without gold-standard labels and to perform model selection from a small set of models.

Strengths

The authors attack a thorny weakness in the weak-supervision literature; the need for a gold-standard dataset. This need has been present going back to the seminal work by Ratner et al. and undermines the approach. It also addresses model evaluation, which is arguably a more important and difficult problem than model training. After all, it's not a real surprise that heuristics could be cobbled together to produce a model. But, quantifying exactly how good that model might be without direct access to labels is another story.

Weaknesses

The paper is somewhat technical and difficult to follow in places. For example, I stared at Thm 2.1 for some time before realizing that Appendix C existed. The writing could be improved by noting connections to the appendices in the main body of the text, where appropriate. In addition, noting the primary tool(s) used to get each result would help the reader intuit the type of machinery being leveraged. Finally, the experiments show some utility of the method. Could the authors give an opinion on the applications or circumstances in which this method could be used for real-world problems? Do we need confidence that the label model is correct? How many weak labels do we need? etc. Appendix A.2 goes some way to helping here. Again, it would be great to mention that this content exists in the main text.

Questions

No additional questions.

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors have addresses the limitations of the work.

Summary

Programmatic weak supervision is a machine learning approach where labeled training data is generated using heuristic rules, domain-specific functions, or other programmatic methods, rather than manual annotation. This technique allows for the creation of large training datasets quickly and cost-effectively by leveraging expert knowledge and automated processes. Uncertainty in programmatic weak supervision arises from the noisiness and variability of labels generated by heuristic rules, domain-specific functions, or automated methods. Unlike manually annotated datasets, programmatically generated labels often contain errors and inconsistencies due to imperfect heuristics that may not accurately capture the true patterns. The paper prososes techniques that estimate Fréchet bounds to validate the performance of programmatic weak supervision models. The proposed algorithms estimate upper and lower bounds for multiple metrics.

Strengths

The algorithm developed in this work has solid theoretical justification, which reinforces its validity and applicability. Additionally, the presented theory is very interesting and makes a significant contribution to the field. Proposing upper and lower bounds for this framework is very useful and interesting, as well as novel.

Weaknesses

Not explaining the problem along with the state-of-the-art issues in the introduction makes it difficult to understand the contributions and the objective of the paper. I believe the paper could be written more clearly; currently, it is difficult to follow. As the authors mention, the labels could be obtained using a label model. If from Section 2.2 onwards a label model is not proposed, but instead the work deals with a dataset where some labels may be incorrect, why not directly state that this is a supervised classification framework with noisy labels instead of PWS? Theorem 2.4 is not easy to interpret. The quality of Figure 1 can be improved, and this figure needs more explanation.

Questions

See Weaknesses Section.

Rating

5

Confidence

4

Soundness

3

Presentation

1

Contribution

3

Limitations

The authors describe limitations in the paper.

Summary

The paper considers the problem of Frechet bound which is the task of determining the infimum and supremum of a function g(X,Y,Z) over the set of joint distribution with fixed marginals. Asymptotic behavior is derived when estimation of condition distribution P_{Y|z} is available with vanishing total variation error. This bound is based on a nice dual formalization of the original optimization problem. This approach is then applied in a weak supervised learning setup to derive confidence interval for the precision and recall. Weak supervised means that cheap label is available which is the random variable Z, and the expensive target label is Y. So if one can estimate the condition probability of Y given Z, the performance guaranties can be derived for label Y which is the target label.

Strengths

I think the dual formalization of the optimization problem is nice part of the paper. I am not sure how novel it is, since the proof heavily relies on some results on previous work. The problem which is considered is relevant.

Weaknesses

It is not clear what is the merit of the approach, see my third question. I am a little bit confused about the contribution of the paper. Estimating the precision/recall seems cheap, but estimating the conditional distribution of the true label conditioned on the proxy label might be expensive? Especially in multi-class setting.

Questions

1) why it is not possible to get high probability results? 2) Regarding Theorem 2.4, if P_{Y|z} can be estimated with an error O(m^{-1/2}) in terms total variation distance (which is the optimal rate for discrete distributions), then the variance of the optimal solution of the problem based on estimates is shrinking of order with the same order? Is that right? 3) Why it is cheaper to estimate the conditional distribution of Y given z, then to estimate the precision and recall of the model based on Y directly?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

I think the this approach can be applied only for small label spaces. In the conclusion, the authors points out that this approach can be applied for finite set, but I believe that this approach has only merit if the label space is very small.

Reviewer Response

Thank you for the clarification! I choose to maintain my score as a weak accept; I think that this paper proposes an interesting approach to a relevant problem in PWS.

Thanks for addressing my concerns.

Thanks for the rebuttal. * Somehow the problem is described better in that paper which is cited [1]. Anyway, I know that every technical detail cannot be described in a NeurIPS due to space limitation. So I also reread your paper, and I think I overlooked the merit of your approach. What I miss is to test various different label models, however, I believe that the confidence interval is hard to derive if the label model is model complex. * This is actually addressed in the conclusion as pointed out that this is a limitation. I think in terms of novelty this paper is already good enough for NeurIPS. * Yes high probability results are conservative, but the asymptotic might be not valid for small sample size. But anyway, do not open this debate here. * Thanks for the comments. So I will increase my score.