Summary
- This work proposes a confidence label based training approach called _ConfDiff_ for binary classification models as an improvement over traditional hard labels. Specifically, authors argue that obtaining hard labels for traditional supervised learning paradigm, or confidence metrics around positive labels for weakly supervised learning paradigm is expensive and at times infeasible process. Therefore, they propose training such classification models on pairwise comparisons between confidence in positive labeling of training data example.
- Authors prove error bounds, and demonstrate the robustness of the proposed approach theoretically.
- Authors evaluate the proposed approach on benchmark classification datasets and a real world recommender system.
Strengths
**Motivation**
Authors make a strong technical as well as intuitive case in support of their approach, _ConfDiff_, in comparison to prevalent approaches in the related work such as _Pconf_. Similar to traditional supervised learning paradigm where approaches such as _soft labeling_, _pairwise loss_, etc. have provided benefits over pointwise models utilizing hard labels in the form of increased robustness and robustness, it is intuitive that pairwise confidence differences should outperform methods like _Pconf_, as demonstrated by the paper.
**Technical Presentation**
The work is very well presented and easy to digest. The Preliminaries section sets up the reader very well equipped to compare/contrast the proposed approach with existing classification approaches. Specifically, the authors do a great job at using formal notations with perfect granularity required to compare different approaches, without being overwhelming to the reader.
**Experiments**
Authors evaluate their proposed approach on multiple benchmark datasets as well one real world recommender system dataset. All the details as well as source code required for reproducibility are available in the Appendix section of the work. The observed evaluation metrics clearly favors the proposed approach against the baseline comparisons. Authors further analyze the sample efficiency and robustness of the proposed approach and provide empirical evidence supporting the theoretical analysis in the prior sections.
Weaknesses
**Choice of comparative baselines**
While the authors clearly position their work as an improvement over the existing approaches in weakly supervised learning domain, they make a case throughout the introductory as well as technical sections regarding benefits over traditional hard label supervised learning. Given that _ConfDiff_ was evaluated using backbones such as ResNet-34, I believe it would help the work to include inside Table 1 the metrics corresponding to vanilla ResNet-34 or logistic regressions.
**Generalizability of proposed approach**
Theoretical contributions not withstanding, it is not entirely clear how generalizable the proposed approach is in terms of obtaining confidence difference values. Authors present the real world evaluation on KauiRec dataset where _watching ratio_ i.e., ratio of watching time of short video and total length of short video, has an intuitive relationship with $p(y=1|x)$.
> We clipped the watching ratio above 2 and regarded the examples with watching ratio greater than 2 as positive examples. Following the experimental protocol of [8], we regarded the latest positive example for each user as the positive testing data, and sampled 49 negative testing data to form the testing set for each user.
As I understand, based on the above text in the Appendix, even in this example, the authors had to resort to assigning hard labels to training data using a heuristic based approach. I am not sure what would be a similar intuitive approach to compute $c(x, x^{\prime})$ in a more common user-item recommender system where purchase labels/click labels are by nature hard labels, and the priors are heavily biased by the fact that only limited number of recommendations can be shown to a user?
Questions
- Could you address the weakness/include discussions about the same in the manuscript?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
In the Appendix section, authors have addressed limitations of the approach being currently limited to binary classification and potential negative societal impact in terms of loss of annotation work.