A central notion in practical and theoretical machine learning is that of a $\textit{weak learner}$, classifiers that achieve better-than-random performance (on any given distribution over data), even by a small margin. Such weak learners form the practical basis for canonical machine learning methods such as boosting. In this work, we illustrate that prompt-based large language models can operate effectively as said weak learners. Specifically, we illustrate the use of a large language model (LLM) as a weak learner in a boosting algorithm applied to tabular data. We show that by providing (properly sampled according to the distribution of interest) text descriptions of tabular data samples, LLMs can produce a summary of the samples that serves as a template for classification and achieves the aim of acting as a weak learner on this task. We incorporate these models into a boosting approach, which in some settings can leverage the knowledge within the LLM to outperform traditional tree-based boosting. The model outperforms both few-shot learning and occasionally even more involved fine-tuning procedures, particularly for tasks involving small numbers of data points. The results illustrate the potential for prompt-based LLMs to function not just as few-shot learners themselves, but as components of larger machine learning pipelines.
Paper
Similar papers
Peer review
Summary
This paper explored an interesting problem that how to apply and extend LLMs over tabular supervised learning tasks. The paper first described each tabular sample as text, and then resorted to LLM to generate the summary for a set of selected representative samples as the template, which can be viewed as a weak classifier. Finally, by integrating these induced weak classifiers followed by booting learning paradigm, a stronger boosting classifier is constructed. Experiments were conducted over several tabular classification data sets to demonstrated the effectiveness of such learning procedure. This procedure makes full use of zero-shot/few-shot learning ability of LLMs and extends such ability to few-shot learning setting over tabular data.
Strengths
1.This paper extends the application scope of LLMs to traditional tabular data and sheds light on integrating LLMs to many real world machine learning systems. 2.The paper gave some effective guidelines to transform tabular samples to text description based on LLMs and metadata automatically with minimal manual engineer. 3.Extensive experiments are conducted to demonstrate some important impact factors on the classification performance.
Weaknesses
1.The current method seems not easy to apply on high-dimensional tabular data (and for high-dimensional data, we might suffer from insufficient training data, also might due to the limited text sequence handled by LLMs) or when there might be many irrelevant features (this might introduce noise in text description). 2.There are generally three types of features in traditional tabular data, quantitative, ordinal and categorical. The paper talks more about numerical features. It is better to give a systematical discussion of how to generate text description over these three different types while considering the meta-data or give the guidelines to generate the prompt patterns over LLMs to get the text description.
Questions
1.In Algorithm 1, the proposed Clustering sampling is based on GPT embedding to group the data. How does the clustering performance? Such as inter- intra clustering distance metrics? Or does such clustering method will group samples with different class labels together? Can we also make use of some encoder based LLMs (such as BERT, BigBird for long sequence) to perform such clustering? 2.How to make use of meta-data of the feature to generate the text description of categorical feature values? And how to generate the summarization prompt pattern based on meta-data? Dose this pattern depend on the property of learning target? For example, binary classification vs multi-class classification. 3.Can we make use of results of tree-based method (such as XGBoost) to guide the preprocess of continuous features? That is, making use of same bin method with XGBoost and remap each bin to a meaningful text phrase based on the feature’s meta-data? This is due to the fact that the performance of proposed method heavily depends on such discretization method. One benefit of XGBoost is that it is able to perform feature selection when generate sub-trees. The irrelevant features of course have negative impact on clustering and summarization. This might be also affecting the conclusion that more examples do not means better performance in “How does the performance scale with more examples?”
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
1.The proposed method might difficult to be directly applied to high-dimension tabular data. 2.The tabular data might be derived from different domains, such as biological? May be we need domain-specific fine-tuned LLMs or pretrained LM to perform clustering or summary to get optimal performance. Therefore, some conclusions especially from ablation study might be LLM depended.
Summary
This paper explores the concept of weak learners, which are classifiers that achieve slightly better than random performance on any given data distribution. The paper demonstrates the effective utilization of large language models (LLMs) as weak learners. The study focuses on applying a large language model to tabular data using a boosting algorithm. By providing properly sampled text descriptions of tabular data samples according to the target distribution, LLMs can generate a summary or template for classification, serving as a weak learner for the task. The paper incorporates these models into a boosting approach, which, in certain cases, outperforms traditional tree-based boosting methods by leveraging the knowledge within the LLMs. The experimental results indicate that the proposed method outperforms both few-shot learning and complex fine-tuning procedures, particularly when dealing with a limited number of data points. These findings highlight the potential of prompt-based LLMs not only as few-shot learners themselves but also as components of larger machine learning pipelines. Overall, the paper showcases the effectiveness of prompt-based LLMs as weak learners in boosting algorithms for tabular data, offering insights into how they can improve classification performance, particularly in situations with scarce data availability.
Strengths
Strengths: --------------- 1. The paper successfully brings together the concept of weak learners in boosting algorithms with the advancements in large language models (LLMs), creating a novel approach for utilizing LLMs as weak learners in tabular data classification. 2. The paper introduces a unique approach by converting tabular data into text form and using LLMs to generate summaries or prompts. This methodology allows the LLM-generated prompts to serve as effective templates for tabular data classification without the need for retraining or fine-tuning the LLM itself. 3. Through comprehensive evaluations, the paper demonstrates that the proposed approach outperforms alternative techniques such as zero-shot and few-shot learning. It also showcases the approach's superiority over traditional tree-based boosting and LLM-based fine-tuning methods, particularly in domains with limited examples. This performance advantage highlights the potential of LLMs as weak learners in boosting frameworks. Overall the paper is well-written and paves a way to utilize LLMs in boosting.
Weaknesses
Weaknesses: ------------------ 1. The paper focuses specifically on tabular data classification, which may restrict the generalizability of the proposed approach to other types of data or domains. It would be valuable to explore the performance and applicability of LLM-based weak learners in a wider range of datasets and tasks. 2. Although LLMs have shown impressive performance in various natural language domains, they still have inherent limitations, such as sensitivity to input phrasing and potential biases in the training data. The paper would benefit from discussing and addressing these limitations to provide a more balanced perspective on the capabilities and potential drawbacks of LLM-based weak learners. What was the rationale for choosing these datasets? Are there some data settings that are favorable to summary boosting and similarly the settings that are not good for the proposed method? 3. Minor comments/questions, a) Please make fonts bold (or use color) in the tables for the best numbers in each row. b) Are the results reproducible? What is the cost of running the experiments?
Questions
Please see the weaknesses above.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
Yes.
Summary
The paper investigates the use of large language models (LLMs) as weak learners in a boosting algorithm applied to tabular data. By providing text descriptions of tabular data samples, LLMs can generate a summary that acts as a template for classification, effectively serving as a weak learner. The authors incorporate these LLM-generated weak learners into a boosting approach and demonstrate their performance superiority over few-shot learning and fine-tuning procedures, especially for tasks with limited data points.The results showcase the potential of prompt-based LLMs not only as standalone learners but also as components of larger machine learning pipelines.
Strengths
1. The experiments are conducted on a large number of datasets and illustrate the effectiveness of the method proposed on tabular data. 2. Clear writing and visualization. 3. The examples in the appendix are enjoyable to read. The prompt templates in the appendix are worth learning.
Weaknesses
1. The integration of multiple weak learners using ensemble learning methods, each requiring the invocation of LLM, may result in significant resource costs. In Appendix A.11, it is calculated that running a dataset of 175 instances would incur a cost of $25, which seems relatively high. 2. The presentation of experimental results could be improved by highlighting the relevant information in the tables. Adding bold formatting to the results would make them more prominent and eye-catching. 3. In Appendix A.12, it is mentioned that ChatGPT performs poorly on some datasets due to the utilization of RLHF. However, no experimental evidence or persuasive argument is provided to support this claim. It is also possible that the issue stems from suboptimal prompts. Also, the Figure 3 shows the results of using GPT-3, where GPT-3 does not consistenty improve upon the smaller model. GPT-3 and ChatGPT have the same pretrained data, which means the performance degradation may be influenced by the pretraining data of the model.
Questions
1. The proposed method to split continuous values in tabular data into discrete attributes and generate corresponding summaries. I wonder whether using this approach to fine-tune the LLM could yield promising results. This way, it can effectively align the tabular data format with the language model while enabling the model to learn latent knowledge embedded in the data. 2. How can the determination of the number of discrete categories (e.g., low, medium, high) for continuous numerical values in a table be made? From what I understand, your approach involves experimenting to determine the optimal number of categories.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
Limitations are discussed in Section 6 of this paper.
Summary
This paper demonstrates that prompt-based LLMs can be used as weak learners, with applications on boosting algorithms for tabular data. By providing text descriptions of tabular data samples, the authors show that LLMs can produce a summary of the samples and use it as a template for classification that can be leveraged as weak learners for the task at hand. The proposed approach outperforms zero- and few-shot learning and, occasionally, even SOTA algorithms.
Strengths
The paper introduces a novel approach to using LLMs as weak learners that can be leveraged on tabular tasks via boosting. The proposed approach appears to be novel, but in order to properly judge the paper's significance, the authors should address the comments below on the quality and clarity of presentation. Compared to zero and few- shot approaches, the proposed Summary & Summary Boosting is clearly superior; however, when compared with KNN, XGBoost, and (especially) TabPFN, the practical applicability seems quite limited and not well understood.
Weaknesses
The paper can be significantly improved by providing a crisper, more intuitive description of the proposed approached (both Summary & Summary Boosting), and by offering an in-depth discussion of the practical impact of the empirical results in Section 4. With respect to the presentation: Figure 2 appears to present the "Summary" algorithm, without Boosting. Even so, it is unclear (i) what is the output of the stratified cluster sampling (I expect it to be a subset of the input to this step), (ii) what does "Select BEST summary" mean - best based on what? only one summary is selected? probably not, because one would expect it to be not so much "the best" but rather "the MOST USEFUL" to classify a new, unlabeled example. Furthermore, lines 137-138 seem to imply that just one summary is chosen (per class?). Adding all these key details to Figure 2 is critical for removing the current ambiguities. The paper should add a new figure, similar to Fig 2, that fully illustrates in detail the Summary Boost algorithm. Last but not least, the authors should also clarify & discuss the results in Tables 1 & 2, for both of which they should BOLD the best result (to improve readability). While it is obvious that both Summary & Summary Boosting outperform zero- and few- shot learners (Table 1), the picture is a lot more confusing when comparing Summary Boosting with the four strong "baselines" in Table 2. First of all, the novel approach obtains the best results on only 3 or 4 of the 16 datasets. Second, often times one of the baselines outperforms the novel approach by almost an order of magnitude (eg, wc & wine). Last but not least, the two conclusions of the paper (that SB does best on datasets with few examples and worst on those with continuous attributes) are not necessarily correct: even though the best performances of SB are on datasets with 1/3/3/5 continuous attributes and 73-306 examples, it is unsafe to generalize from here. For example, iris has 4 continuous attributes and 150 examples, but SB's performance on it is abysmal (0.193 vs 0.027), even though this dataset's properties are similar in nature hams (5 & 73) or tae (1 & 151), where it does comparable/better than the competition. The authors should do an in-depth study about the suitability of the approach to various types of domains; if they need extra space, they could shrink/send-t0\o-appendices Section 5. OTHER COMMENTS: - line 87: you should either explain what "the special token" does, or omit to mention it
Questions
see Weakness above
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
The main concern of this reviewer is the practical usage of Summary & Summary Boost. Without clarifying the issues and questions raised under "Weaknesses," the proposed approach has very limited applicability under still-unclear circustances.
Summary
This paper proposes a novel way to use large language models (LLMs) as weak learners in boosting frameworks for tabular data. The core idea is called LLM Summary Boosting, a novel method that prompts large language models (LLMs) to create weak learners for usages within a boosting framework to make predictions on tabular data. The authors conducted thorough experiments on a variety of benchmark datasets. Compared to conventional tree-based boosting and LLM-based finetuning, this approach demonstrates superior performance in specific scenarios and remains effective with limited examples. To the best of my knowledge, the contribution of this paper is novel despite its incremental nature of contribution. I therefore recommend weak accept and am willing to hear what the authors think.
Strengths
This paper is well-motivated and well-written. I also find the paper very easy to follow along with. Ideas of prompting LLMs to perform data manipulation/wrangling task aren’t new but this paper focuses on the specific usages of prompting for use with boosting scheme on tabular data. I think the contribution is clear here with the experiments.
Weaknesses
While I find the arguments and experiments quite comprehensive, I am personally on the fence about the title, but I do not fully object to it. However, in the current form, the title seems to be a little bit overclaiming.
Questions
N/A
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
4 excellent
Contribution
2 fair
Limitations
The authors described the limitations of the proposed methods in section 6. The authors also provide a detailed failure modes documentation and I think it is very valuable.
Response to author rebuttal
I appreciate the authors' response and would like to share some lingering concerns regarding the utilization of OpenAI APIs for scientific research, primarily centered around issues of reproducibility, ethics, and cost implications. It might be worth considering an alternative title such as "Examining OpenAI GPT-3 as a Weak Learner" unless the scope of the study encompasses a broader range of language models, especially those that are openly accessible. While I find the concepts presented in the paper intriguing, I do have reservations about the evaluation methodology due to its reliance on a commercial API, which could potentially evolve over time. My intention is not to undermine the value of the paper's ideas, but rather to emphasize the importance of a robust evaluation process that stands up to scrutiny and remains valid regardless of potential changes in API availability. From a broader perspective, I am concerned about the overreliance on OpenAI APIs within machine learning research. Paying money to OpenAI to get better numerical results, and associating acceptance solely with better numerical results, can inadvertently limit the overall progress of the field. Personally, I would find the paper's concepts more compelling if the evaluation encompassed open language models that provide full transparency regarding their details and weights. My focus lies more on the soundness and reproducibility of the evaluation rather than pursuing impressive numerical results.
Thank you for the response and engaging in discussion! We agree that the changing API is an important issue that needs to be addressed. **We will add more detailed discussion on this into our paper to emphasize the implication and limitation of using a commercial API**. On the other hand, as far as we are aware, the changes only affected ChatGpt instead of the model through API access so our results should not be affected. You should be able to reproduce the result with the code we provided. The reason why we chose OpenAI’s API access is not for numerical results. As you can see, the work is exploratory in nature and we do not actually always outperform existing methods. The actual reason is that using API lets us test the capabilities of the models at a reasonable efficiency even in an academic setting, as doing the boosting requires many sequential calls to the language model and we need to do it on many datasets in parallel. We fully agree that this is a less-than-ideal solution, but it does enable us to do research that would otherwise be inaccessible to us. At the current pace of innovation in LLMs, we are cautiously optimistic that there will be software solutions (e.g., for efficient inference) that will change the circumstances. Apologies that it took us a while to respond because we were investigating the possibility of using an open-sourced model. Unfortunately, as it stands, we simply do not have the infrastructure to host a LLM with size of GPT3 at a high-enough throughput to run all the experiments in a reasonable amount of time. We will try to reproduce subsets with open-source models such as BLOOM or GPT-J for the final version of the paper. In the meantime, is there anything else that we could discuss or add to the paper that would partially alleviate your concerns?
To address the concerns that our results are only based on GPT-3, we have investigated another LLM - Claude-2 and ran the Summary Boosting method for 4 datasets. The results are as follows. | **Dataset** | **Claude-2 test error rate** | **Curie test error rate** | |--------------------|------------------------------|----------------------| | iris | 0.200 | 0.193 | | caesarian | 0.312 | 0.300 | | vertebra-column | 0.310 | 0.262 | | breast-cancer | 0.385 | 0.288 | As you can see, the performance of Claude-2 is slightly worse than that of GPT-3 (but better than ChatGPT). Note that due to time constraints we have not tuned any prompts for Claude-2 so the actual performance is likely better. This experiment demonstrates that our method still works outside GPT-3, and is able to produce weak learners for different base LLMs. Though the scope of the experiments is limited, it shows preliminary evidence that the method is robust across models. Summarizing documents and text-based reasoning are pretty standard use cases of LLM, so in principle we expect our method to work with other prompt-based LLMs. On the other hand, Claude-2 released by Anthropic is still accessed through a commercial API. We understand that you would like to see open-sourced models but as we mentioned above we do not have the resources to run experiments with these models. We will add the following discussion about the limitations of using commercial API in our revised paper: - **Limited Customization:** Commercial APIs may not allow you to fine-tune the model according to your specific needs. You might not have control over the architecture, hyperparameters, or training data used for the model, which can be crucial for certain research tasks. The backend models offered by these vendors are also continuously updated either through new training methods or more data. For ML research in general, it will be a good practice to mention the version of the model for replicating the experiments. - **Lack of Transparency:** The inner workings and architecture details of commercial APIs are often proprietary and not fully disclosed. This lack of transparency can make it challenging to understand model behavior, biases, and limitations. Specifically it deprives us of the opportunity to study the training process itself, which can be valuable for understanding how these models learn and for improving our own research methodologies. Commercial APIs may inherit biases present in the training data, potentially leading to biased or unfair responses. - **Data Privacy and Security:** Using a commercial API involves sending your data to an external server, which might raise concerns about data privacy and security, especially if you're dealing with sensitive or confidential information. There might also be regulatory and legal considerations when passing sensitive data to these API.
Thank you for doing experiments with Claude-2 and sharing the results. It is understandable to not have the resources to run LLMs locally. I appreciate the discussion on limitations and it would be good to have it in the main paper. I know it's quite late, but would it be possible to run these experiments on some of the quantized versions of LLMs available on Hugging Face? I hope the authors will include these in the main paper and make improvements in the presentation as suggested by other reviewers. I am happy to increase my score.
After authors' rebuttal
Thank you very much for the detailed answer to my review. Your comments helped clear up most of my "tactical comments," but less so when it comes to the "strategic" ones. IMO, the empirical evidence is still too mixed to get this paper out of the "borderline" zone; the situation would be different if the authors had a real-world application domain with clear cut results. Furthermore, the comments of fellow reviewers NJzp and gzhs have also convinced me that my original rating is correct.
Thanks for detailed explanation and clarification on my questions. The response answers most of questions. And I keep my positive rating.
Decision
Accept (poster)