The ALCHEmist: Automated Labeling 500x CHEaper Than LLM Data Annotators

Summary

The paper proposes a labeling workflow (Alchemist) using LLMs where instead of labeling each data point using a teacher model, we ask the teacher model to generate a program to label for the given task. Multiple such programs are generated and using an aggregation function, we get pseudo labels. These pseudo labeled points are then used as training data to train a student model. They show that such labeling mechanism is effective and less expensive compared to the teacher model labeling every data point. They also perform ablation studies for multi-modality, supplementary information during program generation, and diversity of programs.

Strengths

1. Knowledge distillation using a larger LLM to label datasets is very prevalent now. It is an expensive process. 2. Alchemist is an interesting way to reduce costs. 3. The idea to generate weak labeling functions/programs is novel and as the weak supervision literature shows, it can be effective in certain cases. 4. The paper is clearly written, and the experiments section is organized to answer critical questions.

Weaknesses

As the results show, no performance degradation is not guaranteed (for e.g. table 1, Yelp dataset). For a new dataset, without comparing the approaches, it may be hard to understand if Alchemist works for it without impacting the performance.

Questions

1. Why are the accuracy numbers for Alchemist with GPT-3.5 different in table 1 and table 3? 2. How does the performance compare if we use the generated programs for labeling the test set instead of training the student model? 3. Given that the performance gain/no loss is not consistent across datasets, what is the recommendation for a new dataset, i.e. how can one know whether this technique is going to work for that dataset? There are mentions that it works better for complex dataset, any other detailed characterization possible?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The main limitation is that it is hard to understand if Alchemist is going to work for your task without spending the money to label and compare. Performance loss when observed is not small to ignore. Generated programs may be reviewed by human experts, but they may not be straight-forward to interpret and gain confidence from.

Summary

The authors propose an innovative solution for high cost of APIs that using large pretrained models to generate programs that act as annotators. This idea helps replace or supplement crowdworkers and allows for distilling large models into smaller, more specialized ones. Traditional methods can be costly and produce static, hard-to-audit datasets due to expensive API calls. To tackle this, they introduce Alchemist, a system that tasks models to generate reusable labeling programs. These programs can be applied locally, significantly cutting costs while maintaining or even improving performance.

Strengths

1. Generating programs that can produce labels is an interesting idea. This paper introduces a mechanism that can generate multiple labels, transforming the one-to-one mapping between samples and labels into a sustainable one-to-many or many-to-many relationship. This approach can significantly reduce API costs. 2. From a practical implementation standpoint, Alchemist's flexibility and reusability are major strengths. The system allows users to create simple prompts for generating labeling programs that can be stored, reused, and extended. This adaptability makes it a versatile tool suitable for a wide range of tasks and ensures that users can tailor it to their specific needs. 2. The code is straightforward and easy to read, making it accessible even for those who might not be deeply familiar with the underlying concepts.

Weaknesses

1. One of the limitations of Alchemist is that the generated programs tend to handle only fixed tasks, often relying on threshold-based judgments. This means that for more complex or varied tasks, the system might still need to call specialist models via API, which somewhat limits its flexibility. 2. Another weakness is the lack of experimental evidence regarding the stability of the generated programs. For instance, the paper doesn't thoroughly explore how factors like the temperature variable in the code could affect the quality and consistency of the generated labeling programs. 3. The authors could also improve their literature review. There are several highly relevant papers that discuss various aspects of data annotation that weren't cited, such as [1] [2] [3] [4] [5]. [1] https://arxiv.org/abs/2310.04668 [2] https://arxiv.org/abs/2303.15056 [3] https://arxiv.org/abs/2306.04349 [4] https://dl.acm.org/doi/pdf/10.1145/3613904.3642834 [5] https://dl.acm.org/doi/pdf/10.1145/3594536.3595161

Questions

Refer to weaknesses.

Rating

6

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

No

Summary

The paper presents a new method for creating data labels that leverage a Large Language Model and the weak supervision/data programming labeling paradigm. In this work, the LLM is used to generate labeling code, typically in the form of functions, which can then be used to create weak labels for weak supervision. The paper goes on to show this method’s utility across several benchmark text datasets and one image dataset. The paper also does an ablation study showing the impact of different elements of the method, like numbers of weak functions, and including examples in the prompt to create the functions.

Strengths

The paper is strong in its significance, clarity, and quality. For significance, the paper is attacking a significant problem. One of the great promises of large ML models, with multimodal or text only, is their ability to zero-shot label text. This ability allows LLMs to possibly overcome one of – if not the chief issue – with building machine learning models: labeled data. Thus, this paper is attacking a profoundly important problem for the application of ML to real-world problems. For clarity, the paper is well-written and the diagrams are very helpful. When combined with the appendices and supplementary material, I can not only easily see how to reproduce their results, but how to apply this method to my data labeling problems. In other words, I can easily see wide adoption and use of this method. Finally, the paper does a reasonably good job in its empirical testing to cover many variations on the application of the method (e.g., having both text and image datasets) and variations to the method (e.g., including examples, numbers of weak functions, etc.).

Weaknesses

The weakness of the paper is in its grounding/novelty and some of its methods, particularly when applied to other modalities. For the grounding, while the paper actually does capture many of the previous works that have done something similar, there are works like Cruickshank and Ng, “DIVERSE: Deciphering Internet Views on the U.S. Military Through Video Comment Stance Analysis, A Novel Benchmark Dataset for Stance Classification” where they used LLMs + weak supervision to create an actual new dataset. They did not, however, have the novel insight about creating the weak labeling functions that this work does. For the method in the image or multimodal case, two works have done something very similar with CLIP. First, Adila et al. “Zero-shot Robustification of Zero-shot Models” use the same CLIP model and water birds dataset, but get substantial performance increases by “debiasing” the image embeddings, with text characteristics. This is very similar to what is done with the labeling functions where this paper tries to get labeling functions to classify aspects of the birds and ignore spurious contexts. Second, Bendou et al.’s “LLM meets Vision-Language Models for Zero-Shot One-Class Classification” presents a method for one-shot classification using VLMs, which uses an LLM to build negative classes around a positive class. This is similar to the insights in this paper around developing labeling functions to highlight important visual distinctions in the classes to improve the labels. Taken together, I think this paper might be able to incorporate the insights from these other papers to actually improve their results. For example, the debiasing in the first paper could be used with the proposed method.

Questions

I have no additional questions.

Rating

8

Confidence

4

Soundness

3

Presentation

4

Contribution

4

Limitations

One other societal limitation the work could address is how works like this are changing the data labeling industry. Data labeling is still a massive, multi-million-dollar industry and these works are changing the nature of that industry. Ideally, they are changing it positively, but it can still disrupt how the data labeling industry works and cost human labelers their income.

Summary

The paper proposes an automated way to label large quantities of data by leveraging large language models to generate labeling functions which can be used to label data using programmatic weak supervision. The paper demonstrates that this procedure can generate labeling functions which are more accurate than manually generated LFs and querying LLMs for labels directly at a fraction of cost. The authors also show that prompt tuning benefits their method, and that it can be used for other richer modalities.

Strengths

1. The paper is well written, and easy to follow. 2. The experiments are well designed. 3. The idea is simple, intuitive and has promising performance. 4. Evaluation on richer data modalities is challenging, and I appreciate the authors' inclusion of image datasets.

Weaknesses

Most of these concerns are minor, and some of them can be readily fixed. 1. **Comparisons with related work:**: I believe that comparisons with empirical ScriptoriumWS and DataSculpt are necessary to demonstrate the benefits of the proposed approach. With that said, I found the experiments to be rigorous. I would also like the authors to catalog the differences between their approach and ScriptoriumWS, DataSculpt, and [1] to better highlight the novelty of their methodology and findings. I am aware that [1] was published very close to the NeurIPS deadline so I do not expect the authors to compare their methods with [1], but conceptual differences (if any) can still be highlighted. 2. **Missing references:**: The idea of using keywords and LLMs to automatically create LFs is not new, so I would encourage the authors to reference some prior work (e.g. [2]). 3. **Lack of reproduciblity and claims:** The techniques presented in the paper and the results are not reproducibile with the details given in the paper. I would encourage the authors to release their code for the same. The paper mentions that "any particular program may be inaccurate, fail to compile, or may otherwise be flawed, ...". I am curious how the authors ensure that the programs are not downright inaccurate (because snorkel would require the accuracy of LFs to be greater than random chance), or fail to compile? I believe that some kind of a data-driven feedback loop is important to ensure that the programs compile, and the generated LFs have coverage, and are accurate. Therefore, I am excited about the possibility to include keywords, dataset descriptions, example in the prompt, but I feel that there has to be a feedback loop after the label model labels the data or identifies inaccurate LFs. 4. **Richer modalities:** The claim on scaling to richer modalities should be softened, because the authors only evaluate their approach on a relatively simple image dataset. More challenging tasks may include labeling chest X-rays [3], datasets for which are publicly available, and other richer modalities such as time series data [4]. I do not expect the authors to conduct these experiments during the rebuttal, but such an extension would be very interesting and valuable, in my opinion. ### References 1. Smith, Ryan, et al. "Language models in the loop: Incorporating prompting into weak supervision." ACM/JMS Journal of Data Science 1.2 (2024): 1-30. 2. Gao, Chufan, et al. "Classifying unstructured clinical notes via automatic weak supervision." Machine Learning for Healthcare Conference. PMLR, 2022. 3. Irvin, Jeremy, et al. "Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison." Proceedings of the AAAI conference on artificial intelligence. Vol. 33. No. 01. 2019. 4. Goswami, Mononito, Benedikt Boecking, and Artur Dubrawski. "Weak supervision for affordable modeling of electrocardiogram data." AMIA Annual Symposium Proceedings. Vol. 2021. American Medical Informatics Association, 2021.

Questions

I wonder what the authors thoughts are on points 3 and 4 above in the weaknesses.

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed the limitation of their approach.

Thank you for the rebuttal!

Dear Authors, Thank you so much for your rebuttal and putting in time and effort. I have raised my score to reflect my current assessment of the paper. Below are some general thoughts based on the rebuttal: 1. In addition to data-driven feedback, I also think that output driven feedback is important. In some cases the LLM might hallucinate and not generate LF code grounded on the input prompts, in which case an output-driven feedback loop can ideally guide the model to generate correct programs (accuracy is the second step, the model must generate correct programs first). I foresee this to be a problem in non-text modalities where the model is having to use APIs or local feature extractors. I think some discussion on this and your empirical findings would be beneficial to the community. 2. "*Both ScriptoriumWS and DataSculpt address weaknesses in weak supervision, whereas Alchemist focuses on the downsides of model-based annotation.*" Can you explain this statement? 3. It seems that ScriptoriumWS and Alchemist are similar in terms of accuracy and the # of programs which are generated. It seems from your description that the difference lies in extensibility to other richer modalities. Can you explain what exactly prevents the core ScriptoriumWS methodology to model richer modalities? The answer can be as simple as, "the method *can* be extended with X, Y and Z, but this was not evaluated". 4. I would like to authors to provide some details about their time series classification experiment. The results are encouraging, and this is out of curiosity. I re-read the section on richer modalities in the paper, and the extension to time series doesn't seem trivial. I would like to know more about how the authors carried out this experiment.

Reply to Rebuttal

The Authors have improved the paper further. I am particularly impressed to see the Roboshot idea incorporated and that it further enhanced the method. I stand by my rating of this being string accept.