Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models

Summary

The author presents a complete and thorough study in their research regarding semi-supervised learning of social science application targeting bias and coverage control. The work include clear mathematical proof of the purposed approach, and experimental results demonstrating the well alignment of theory and application. The assumption is also clearly stated and limitations are considered. As a non-expert of computational social science, the paper is relatively easy to read and understand. Overall I think this is a solid work in the field. I have a few minor questions and comments below.

Strengths

1. the mathematical proof of bias correction step, equation 4 and the conditionally unbiased estimation of Yi is simple and clear 2. the surrogate labels can be arbitrarily biased, while the theoretical guarantee still holds is the most fascinating and important contribution of the work. I personally find this really interesting 3. experimental results show great alignment of theory, comparable performance of RMSE compared with SSL although the method does not specifically target for it

Weaknesses

1. the biggest miss/weakness after reading from my perspective is from the title, using LLM annotations for valid statistical inference. In general the research is more like a mathematical proof of bound on semi-supervised learning and demonstration of usefulness with experiment. There is little regarding using LLM annotations. With such a title I would expect a substantial part of the work focusing on LLM generated annotations, e.g. how it is used and how can it be improved from LLM perspective, etc. 2. the assumption of the purposed approach, though clearly stated, greatly limit the contribution of the approach. It may be a common setting in social science that test set is known, but not the case for majority of LLM annotation use cases. 3. the asymptotic behavior given the gold-standard annotation, is not well discussed in the main paper. Aspects like the downstream coefficients expectation and variance given the size of golden labels compare with sample space size are not well covered. Some empirical numbers could be extremely useful, beyond the settings in experiment part 4. some minor comments: page 8 figure 2 annotations, as a non expert of social science it would be good to explain in more detail what is coverage and why it is important to the study. In second experiment class prevalence estimation, better include more details like each dataset's distribution, total size of each dataset, etc. (see it in supplement however I think it is helpful to include some numbers in main paper, as it gives the important size ratio of golden-set and total set)

Questions

Please see comments in weakness part

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

the limitations of the work is well discussed.

Summary

In order to exploit the advantages of Large Language Models (LLMs) in computational social science (CSS) area, this paper proposed a novel DSL estimator, which employed a doubly-robust procedure to combine surrogate labels with a few gold-standard labels. By using the proposed method, not only LLMs can be used as surrogate label generator, but also the asymptotic unbiasedness and proper uncertainty quantification in CSS can be maintained. Finally, this paper conducted plenty of experiments to prove the effectiveness of the proposed method.

Strengths

1. This paper proposed a unified framework for using imperfect surrogate labels in downstream statistical analyses which maintains the CSS priority for unbiasedness and proper coverage. 2. This paper proposed a strong theoretical guarantee for the effectiveness of the proposed method. 3. This paper conducted extensive experiments across 18 datasets to compare the performance of the proposed method.

Weaknesses

1. This paper focused on the semi-supervised setting in some CSS scenarios. The difference between the proposed method and previous semi-supervised methods should be compared in a detailed way. 2. This paper had plenty of notations. An explanation table about these notations should be provided. 3. In experiments parts, this paper should provide a brief about the evaluation tasks and corresponding datasets, so that the results can be more convincing.

Questions

N/A

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please see the weaknesses for more details.

Summary

This paper studies the use of surrogate labels in model training and it's bias, coverage and performance properties. Practically, the premise is to use a small number of gold data annotations for a classification task, then construct surrogate labels that have desirable bias, coverage and performance properties. The application of this would be doing additional analyses using covariates that would benefit from the additional support in data size. The method proposed uses a small amount of gold labels to calibrate the model prediction and is agnostic to the choice of underlying model. The method is compared to using gold labels alone, surrogate labels alone from an external model not fitted to the data and a simple form of semi-supervised learning where a model is fitted to the gold labels. Experiments are conduced in depth on bias, coverage and RMSE in more depth on one data set on congressional bills. Further experiments are conducted over a battery of data sets related to computational social science applications on bias, coverage and RMSE with 100 gold labels. The results show that bias and coverage for the proposed method are closer to using gold standard data alone, while showing small improvements on some datasets in RMSE.

Strengths

The premise and application is interesting and has not been too well studied to the best of my knowledge. A large number of data sets were used in the bias/coverage/RMSE experiments, which provides a good starting point for such analyses. The proofs and experiments supporting that bias and coverage properties of the proposed method.

Weaknesses

The key weakness is that application presented as a motivation for this paper is not tested throughout any of the experiments. The covariates are not used in any of the experiments to enhance data analysis, which could have been done with several data sets in the collection, including the congressional bills. The experiments show minor improvements in general on the RMSE, especially when compared to the vanilla self-supervised methods. The self-supervised methods represented here are very weak for this category of methods. For example, the gold labels are overwritten by the self-supervision and the self supervision is done in a single round over the entire data set, without any filtering by confidence as usually done, which makes this an artificially weak baseline, especially on bias and coverage. Yet, the SS methods achieve constantly higher RMSE, especially when considering the difference between Gold labels and the proposed method (DSL), which shows that the DSL methods does not capture much of the performance benefits that self-supervision alone could achieve. This leaves the main application of the approach the ability to use more data for covariate analysis, but this application is now shown through experimentation in the paper. The paper should also present results in F1 metric (both surrogate label model and method comparison results) because RMSE is not natural for multi-class classification and having the surrogate label model and methods trained on top of predictions on the same method is useful for comparisons. As a general comment: the method is agnostic to the source of surrogate labels, as any model can be used to produce the labels and the proposed method does not use any information about the model or its assumptions. Hence, while LLMs can be a source for labels, the paper should be rewritten to remove so many specific mentions of LLMs and discussions about LLMs, which serve no purpose for the paper (perhaps it would for paper publicity), but actually artificially narrow the generality of the method. The paper also cites many papers that make unfounded claims or estimates about accuracy needed for 'annotations' (surrogate labels) for applications or model training, while none of the cited papers (e.g. Gilardi 2023) actually trained models to test the improvements those could bring (or not).

Questions

See weaknesses. An interesting analysis would be to use either synthetic data or to intentionally vary the accuracy of the underlying model on the same data set (rather than existing accuracy which differs across data sets). This would provide a more controlled analysis of these factors. What is the K (defined in line 192) in the experiments?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

NA

Summary

The authors consider using outputs of LLMs for labeling in downstream analysis in computational social science while guaranteeing statistical properties (asymptotic unbiasedness and uncertainty quantification). They show that directly using LLM-produced labels yields bias and invalid confidence intervals, then propose the design-based semi-supervised learning (DSL) estimator as an alternative. DSL combines surrogate labels from LLMs and gold-standard labels within a doubly robust procedure that guarantees valid inference even when the surrogates are arbitrarily biased and without requiring stringent assumptions. Experiments on 18 datasets show that DSL yields valid statistical inference while achieving error rates comparable to existing alternatives that focus on prediction with statistical guarantees.

Strengths

The paper is very well written, the setting and methodology are well described and motivated. The assumptions and potential use cases are clearly stated. Further, though the proposed approach is conceptually simple, it blends sound ideas from semi-supervised learning (not necessarily in the, transductive, machine learning sense), doubly-robust estimation, K-fold cross-fitting and automated labeling with LLMs.

Weaknesses

Perhaps the biggest weakness of the paper is that, though to the authors' admission, it prioritizes unbiasedness and coverage, however, DSL does not outperform GSO which is a very simple approach that does not require anything that is not available (a method for surrogate labels and the estimation of g). As a result, it is difficult to imagine a situation, more so in social science, where someone will prefer DSL over the far simpler GSO, unless the reviewer is missing something. As minor points: i) the title of the paper is somewhat misleading because, as the authors point out, the proposed approach is agnostic to the methodology used to obtain the surrogate labels, so the fact they use LLMs is more of a detail of the experiments than a characteristic or contribution of the proposed approach; and ii) the paper (in its current form) does not seem a good fit for a conference proceeding considering the amount of important experimental details that need to be relegated to the supplementary material, without which, it is extremely difficult to properly understand the results.

Questions

In several places, the authors emphasize the need for knowing \pi (the probability of obtaining a gold-standard label) and \pi > 0 for all samples. However, this is not further discussed or described in the experiments. For instance, in Section 2.2 in the Appendix, the authors claim that \pi is known because they can choose which samples to label, however how does that make \pi for all samples that are not going to be labeled (with probability 1) larger than 0?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors discuss some of the limitations of the proposed approach. However, the limitations are quite generally about the addressed setting and less so about the methodology. Also, they point to four limitations, however, only three are described (the particular setting, access to gold standard labels and the focus on bias and coverage rather than MSE).