Summary
The paper focuses on the bias issue in weak supervision. The paper shows the labeling functions of weak supervision may produce biased pseudo-labels, which is first empirically shown in the paper. Also, they theoretically show that even though a dataset has fair ground-truth labels, the weak supervision process using labeling functions may produce biased weak labels for the dataset. To mitigate this issue, the paper proposes an optimal transport-based algorithm that modifies the weak labels to be fairer. In experiments, the paper evaluates the proposed algorithm in tabular, NLP, and computer vision datasets.
Strengths
S1. The paper considers an important research problem, the fairness issue in the weak supervision pipeline. It seems this paper is the first work to handle fairness in weak supervision.
S2. The paper theoretically shows the labeling functions may produce unfair pseudo-labels even though the underlying label distribution is fair.
S3. The proposed method can be used together with the existing fair in-processing algorithms, which is another good aspect of the paper.
Weaknesses
W1. The explanations on the labeling function itself are limited.
- For example, can the number of labeling functions affect the fairness performance? The Snorkel original paper [1] mentioned that the number of labeling functions highly affects the labeling performances, but the current paper does not explain how such details may make differences in the fairness scenario.
- Also, as shown in the vision dataset experiments, using fairer labeling functions can reduce the effectiveness of the proposed algorithm. Then, why is the proposed transport algorithm better than making labeling functions itself fairer? Currently, the paper seems to use very simple labeling functions (described in the appendix), and it would be helpful if the paper could provide any comparison between the proposed algorithm and another possible direction of making fairer labeling functions.
[1] Ratner et al., Snorkel: Rapid Training Data Creation with Weak Supervision, VLDB’18
W2. Experiments show some questionable results.
- As several scenarios show large F1 score drops, it is unclear whether it is okay to use the proposed algorithm. I understand the F1 score can be affected by the imbalanced label classes, but the F1 score degradation is still severe. For example, SBM (OT-S) + LIFT case in the Bank Marketing dataset and all SBM results in the Civil dataset show large F1 score drops. It would be helpful if the paper could provide at least some possible way to prevent such F1 score drops.
- In the experiments, no error range is provided, which makes the empirical results less convincing.
Questions
Although the paper provides several meaningful discussions and insights, there are some remaining concerns, especially regarding the labeling function and empirical results. The details are in the above weakness section.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The paper did not discuss the limitations and potential negative social impact of the work. A possible limitation of this work can be other prominent fairness definitions that this work cannot handle.