Tabular Transfer Learning via Prompting LLMs

Summary

This paper proposes a novel framework, Prompt to Transfer (P2T), which utilizes large language models (LLMs) for tabular transfer learning. The methodology leverages in-context learning (ICL) by creating pseudo-demonstrations from unlabeled or heterogeneous source datasets to predict target tasks effectively. The paper is well-structured, offering a clear introduction to the problem and a detailed explanation of the approach. The experimental results show that P2T significantly outperforms existing methods across various datasets, to support the significance of the work.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

- The P2T framework introduces a novel use of pseudo-demonstrations for tabular transfer learning, which is a significant contribution to the field of machine learning with LLMs. - The experiments conducted show that P2T consistently outperforms traditional and contemporary methods in zero-shot and few-shot learning scenarios. This is clearly demonstrated with extensive experimental validation. - The research addresses a real-world problem of data scarcity in tabular datasets, making the findings relevant for various applications, including finance and healthcare.

Reasons to reject

- Some aspects of the methodology, particularly how the pseudo-demonstrations are generated and utilized, could be described more transparently to enhance the reproducibility of the results. - There is insufficient discussion on how the selection of source and target datasets might affect the generalizability of the P2T framework, which could be a concern for adopting this method in broader applications.

Summary

This paper presents a technique called P2T for transfer learning in tabular data. The core idea of the technique is as follows: supposed you want to use LLMs for doing few or zero shot learning in a domain T where the target prediction is column t, and you have access to data from related but perhaps not exactly the same domain S. First you ask an LLM to identify the column in S that is most relevant to t; let's say s. Then you create few shot examples from S where the target prediction is column s. Each of these examples is essentially descriving the row in natural language. Additionally you can also add few examples from domain T. This makes your prompt. Using this prompt, you can ask the LLM to predict the value of t for any incoming example from T. The main claim of the paper is that this technique improves prediction in domain T and outperforms techniques like LiftICL. In paticular the claims of the paper are 1. When asking the LLM for most relevant target column s, and including those examples in the prompt, improve 1 shot (i.e. 1 example per example from target domain) accuracy of the model, even if S is heterogenous from T. 2. Which column is selected for transfer from S, matters. Performance drops if you take a random column as target. 3. Including unlabeled examples from domain T also improves the performance.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

1. The paper provides a simple and useful idea for transfer learning when using LLMs for tabular data prediction. The presented approach is simple, even incremental, but still useful. 2. The experiments for the most part are useful, and the results for 1-shot heterogenous transfer is particularly quite nice.

Reasons to reject

1. The openml data is particulatrly simplistic. Could the authors also try to replicate this work on TabZilla which seems more suitable for comparisons between various LLM techniques as well as LLM vs other ML techniques. https://openreview.net/pdf?id=CjVdXey4zT 2. Given the surpising nature of the outcomes, I do want to hold the bar on release of prompts and outcomes used for comparing p2t vs lift-icl e.g. so the community can help spot check the outcomes, and since it is trivial to verify the outcomes.

Questions to authors

1. I am a bit confused by the section on including unlabeled examples. In what way are they included in the prompt? Why can't they also be included in LIFT-ICL and how does P2T even differ from LIFT-ICL in this case (since there is no source domain so to speak)? The finding that including unlabeled examples in the prompt helps has been brought up elsewhere as well (https://arxiv.org/pdf/2404.11018) and seems orthogonal to the core p2t idea itself. What am I missing here? 2. What does inclusiion of LR and kNN in table3 mean? is LR trained on source and target data combined and if so, how does transfer happen in this case (is it via just common regularization)? If so, it is perhaps better to follow the tehnique here: https://arxiv.org/pdf/0907.1815. 3. The finding that adding heterogenous source improves the performance significantly is quite surprising. I'd highly encourage the authors to release all of their prompts for this finding for independent verification (which should be trivial). 4. How does the performance change with the closeness between two datasets? What happens if the tasks are far apart? 5. If I were allowed to fine tune a model on source examples, how does the performance change? 6. How sensitive is this approach to different naming styles or conventions across the data?

Summary

This paper proposes a framework called "Prompt to Transfer" (P2T) for performing transfer learning on tabular data using large language models (LLMs) and in-context learning. The key idea is to create "pseudo-demonstrations" from unlabeled or heterogeneous source data that are relevant to the target task, and then prompt the LLM with these demonstrations along with a few labeled examples from the target dataset. Specifically, P2T first identifies the column feature in the source data that is most correlated with the target task label. It then constructs pseudo-demonstrations where that column is treated as the prediction target using the other columns as input features. These demonstrations capture knowledge about predicting the correlated column, which can transfer to the actual target task. Experiments on various tabular datasets show P2T consistently outperforms previous methods in few-shot and zero-shot transfer learning settings.

Rating

5

Confidence

4

Ethics flag

1

Reasons to accept

- Tabular data is very common across domains like healthcare, finance, etc., but tabular transfer learning has received little attention compared to areas like vision and NLP. The P2T framework is specifically designed for tabular datasets. Developing effective transfer learning methods for tabular data is a significant challenge due to heterogeneity across datasets. P2T tackles this by leveraging large language models and in-context learning in an innovative way. - Annotating tabular data is expensive. An effective transfer learning solution that can leverage unlabeled/heterogeneous sources can have substantial practical impact across many data-rich fields.

Reasons to reject

- The paper is largely empirical in nature and lacks deeper analysis of the proposed P2T framework. Without principled justifications, the work may be perceived as somewhat ad-hoc or heuristic. - The experiments mainly focus on classification tasks (mostly binary classification) across a limited set of 17 tiny tabular datasets. Evaluating P2T on more datasets, especially larger-scale ones, as well as other tasks like regression could improve the generalizability of the results.

Questions to authors

1. In the pseudo-demonstration construction, how sensitive is P2T's performance to the choice of the correlated column feature identified by the language model? What happens if the wrong column is selected? 2. The pseudo-demonstrations are constructed based on the assumption that predicting the correlated column feature is a good proxy for the original target task. However, what theoretical justification or empirical evidence do you have that this assumption holds across different types of tabular datasets? Are there scenarios where this assumption could break down? 3. The current approach relies on enumerating and scoring all columns to identify the most correlated feature. How does this scale to datasets with hundreds or thousands of columns? Have you explored more efficient strategies? 4. What about handling missing values or noisy data entries commonly presented in tabular datasets? How can the framework be extended to be more robust and handle realistic data issues?

Summary

The paper presents a novel framework, Prompt to Transfer (P2T), which leverages large language models (LLMs) for transfer learning in the domain of tabular data. The authors address a significant problem in machine learning applications where labeled data is scarce, and they propose a solution that creates pseudo-demonstrations from unlabeled or heterogeneous source data to enhance the learning process. The paper claims that P2T outperforms existing methods on various tabular learning benchmarks.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

1) The paper introduces a unique approach to transfer learning in tabular data by utilizing LLMs, which is a novel contribution to the field. 2) The authors provide extensive experimental results demonstrating the effectiveness of P2T across diverse datasets, which supports the significance of their work. 3) The method's ability to operate without the need for training makes it accessible to environments with constrained hardware resources.

Reasons to reject

1) The proposed method in the paper is too simplistic and does not fully exploit the capabilities of large language models. There is a lack of exploration into the characteristics of tabular data. 2) The method in the paper requires calling third-party APIs, which raises privacy concerns in real-world applications. It is suggested that the authors consider using open-source large language models. 3) While the effectiveness of using Pseudo-demonstrations has been verified, there should be more attempts and comparative analyses in constructing Pseudo-demonstrations. 4) The section on Correlation Identification needs to experiment with conventional algorithms to demonstrate the superiority of using large language models.

Questions to authors

1) How many test samples are there in each dataset? 2) The experiments in the paper only involve the case where the transfer source is a single dataset. Is it feasible to use multiple datasets?

somwhat helpful; rating remains unchanged

Your clarifications do help but some things are still not clear 1. The lr/knn baselines are still not super clear and I believe they have not been optimized to the fullest extent possible for transfer learning. 2. For unlabeled examples: can you provide an example of how it is used? e.g. let's say I have columns x,y,z and target column t. For unlabeled examples, column t is empty. So now you find the most correlated column to 't' amongst x,y, and z? If so, what is this column for the datasets being discussed? The presence of such a column in features does not seem like a very likely phenomenon.

Thanks!

Dear Reviewer ijuV, We truly appreciate your acknowledgment of our rebuttal and overall positive review! We are happy to hear that our response helps to address your suggestions. In the final draft, we will add the clarifications, which we believe will further improve our manuscript. If you have any further comments and suggestions, please let us know. We are committed to improving the quality of our work, and we value your feedback. Thank you very much, Authors