Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering

Summary

The paper proposes CAAFE, a Context aware automated feature engineering approach to support auto ML by utilizing Large Language Models (LLMs). Proposed method generates new features or transforming the feature space from tabular data by using LLMs and incorporate them to train new models. Transforming the feature space can be an iterative process where in each step LLM generates code that is run on training and evaluation sets, followed by fitting a new model and evaluating its performance. LLM receives a set of instructions through a prompt which includes semantic information about the data set, feature names, their data types, useful stats about features, and a few sample values. The paper reports experimental results on two sets of data sets: 1- OpenML data sets published before Sept 2021 (GPT 3.5 and 4 training date); 2- Kaggle data sets published after Sept 2021. Results show that CAAFE can improve the performance of a state of the art classifier. They also show that CAAFE can help build models that outperform models built with the help of other state of the art automatic feature generation.

Strengths

Although the seminal idea proposed in the paper is rather straightforward, it's novelty makes it a worthwhile body of research for consideration. - originality: The work is novel and original. It introduces the use of LLMs for auto ML through adding contextual information which is less explored. - quality: The paper is technically sound and experimental results support the claims. Limitation and risks of the work are also clearly called out. - clarity: The paper is easy to read and follow and the contributions are discussed clearly. - significance: Experimental results show LLM's usefulness for feature engineering aspect of AutoML which testifies the significance of the work. For more detailed comments on submission's significance refer to the limitation section.

Weaknesses

- Alternative feature generation methods that are considered in the experiments are limited. AutoFE is the main alternative method. Other techniques such as PCA or deep sparse feature learning (authors have mentioned that with small data sets they don't expect much benefit from deep learning methods but it would be interesting to emperically validate this hypothesis for the studied data sets) is not considered. - The human-in-the-loop aspect of CAAFE is not fully discussed. For example, one of the implications of having human in the loop is correcting the potential mistakes that LLM may make. In the paper it is not discussed how many times the generated code by LLM needed to be revised by human, not necessarily due to run time errors but because of the semantic issues in the generated features.

Questions

- How do you make sure the generated feature makes sense? e.g. https://github.com/cafeautomatedfeatures/CAFE/blob/main/data/generated_code/airlines_v3_0_code.txt why Euclidean distance between airports even make sense? Based on the data set description it is not clear if from or to airport attributes represent distance to a canonical point. - Minor, line 195, please fix the question marks in "Table ??".

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The main limitation of the work is stated by authors as "We focus on small datasets with up to 2,000 samples in total, because feature engineering is most important and significant for smaller datasets". As stated, the proposed approach is mainly beneficial for smaller data sets which makes the method less useful for a lot of real world and specially industrial applications. Along the same lines, as clearly mentioned by author(s), another limit of the work is the possibility of exceeding the prompt maximum length.

Summary

This paper presents Context-Aware Automated Feature Engineering (CAAFE) for integrating domain knowledge into the AutoML process using LLMs. CAAFE automates feature engineering for tabular datasets, generating Python code that generates semantically meaningful features based on the dataset and a textual description of the data. The study demonstrates the effectiveness of CAAFE on a range of benchmark tasks.

Strengths

- Well written and easy to follow. Thank you! - Simple and very applicable to real-world scenarios!

Weaknesses

- If set up correctly, the method really can't fail. Features are only added, if they improve the objective. Thus the scientific value is somewhat limited to showing that LLMs can create features that improve performance (expected) and the extent to which (here is where I see most value). Therefore comparison to other feature engineering methods seems really important and could be more extensive. - Fig 2 is not clear. Is this a flow from left to right? How does the bottom part relate to the boxes? Please explain better. Suggestions: - I don't think the phrase "code as interface" is self-explanatory. Maybe can be introduced explicitly - In section 2.1., the sentence "GPT-4 is a deep neural network that uses a transformer architecture" - may not actually be correct. It is rumoured that GPT4 consists of multiple models and uses some mixture of experts methodology. In any case, this information is not published which might be worthwhile pointing out. - l195, table reference is missing

Questions

- What about the dataset where performance does not improve or even weaken? Is there an explanation.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

2 fair

Limitations

- The main limitation is more extensive study of competing approaches. In essence the authors work really doesn't answer any open scientific questions.

Summary

In this paper, the authors propose a feature engineering method that builds upon LLMs. Features are engineering in an iterative process of prompting the LLM to generate code for new features, evaluating the features with a ML model and generating a new prompt. Hence, the overall approach follows the common wrapper approach. In their empirical evaluation they find feature engineering via LLMs to outperform no feature engineering.

Strengths

- To the best of my knowledge, this is the first work to employ LLMs (GPT) for automating feature engineering. - The general idea seems to work out quite nicely and compare comeptitively to other feature engineering methods and preferably over engineering no features. - The paper is well written and easy to follow. The overall presentation is really excellent - maybe due to the use of ChatGPT for the editing?

Weaknesses

Although the idea in general is quite interesting, I have several doubts regarding this work. The first and foremost doubt is about how to sort this work into the already existing literature and whether it really establishes a new state of the art. In particular, I am thinking about the following two works which have been missed also during the discussion of related works: Li, Liyao, et al. "Learning a Data-Driven Policy Network for Pre-Training Automated Feature Engineering." The Eleventh International Conference on Learning Representations. 2022. Zhang, Tianping, et al. "OpenFE: Automated Feature Generation beyond Expert-level Performance." arXiv preprint arXiv:2211.12507 (2022). (https://openreview.net/forum?id=1H1irbEaGV) This might be due to these works being rather recent. However, preprints of these papers have already been available for some time. I believe a comparison to these methods is inevitable to prove the real benefit of LLMs and whether the semantic information of feature names is really needed to come up with better features. Another issue with the paper is that it claims that LLMs are taking advantage of the semantical information of a dataset, e.g., the names of the features etc. But this is not substantiated in the experiments nor visible in the explanations generated in the examples. Moreover, I am a little bit doubtful to what extent the explanations of an LLM can be meaningful when LLMs base the text generated on the likelihood of the next word. In particular, logical argumentation does not seem to be a strengths of GPT so far and the flaws in logical reasoning can be made easily visible. E.g. testing with a sentence like "The [doctor/nurse] yelled at the [doctor/nurse] because she was late. Who was late?" and asking for an explanation for the decision. Why should the explanations be more meaningful for datasets? Speaking about the semantics that are supposedly extracted from feature names, it would have been interesting to see whether this information is indeed used, e.g., by blinding feature names. Neither in the main paper nor in the supplement I could find experiments that aim at answering these questions. However, these experiments lacking mean an unproved claim made in the paper. Minor: - Reference to table in line 195 broken.

Questions

1. How does CAAFE compare to the SOTA in automated feature engineering? 2. What makes CAAFE semi-automated? As far as I understood, CAAFE only requires an input by the user once in the beginning to give a description of the dataset. If this is the reason for semi-automation are not every AutoML tools out there semi-automated? 3. Do LLMs in CAAFE really leverage semantic information about the dataset? 4. As the authors already state in their Broader Impact Statement there is a certain risk that biases contained in LLMs transfer to the engineered features and that CAAFE then builds features based on these biases. Should not every feature engineering approach based on LLMs directly incorporate mechanisms to prevent to leverage such biases? Especially since such biases might be very subtle.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

4 excellent

Contribution

3 good

Limitations

The authors prove a broader impact statement estimating what effect and potential negative societal impact there might be.

Summary

This paper presents a novel approach to automated feature engineering utilizing large language models (LLMs). Authors propose to use LLMs to generate code for feature generation. In the proposed method CAAFE, LLM is given a prompt with dataset description and a task of writing code for feature generation, the new feature is accepted if the validation score improves after training an ML model on a dataset containing the new feature. This procedure is repeated. The paper demonstrates the effectiveness of the approach by testing on 10 public datasets from OpenML and 4 datasets from Kaggle (less likely to leak into the LLM training set) and slightly improving the overall classification accuracy of the strong tabular classification model TabPFN

Strengths

- I find the idea of incorporating LLMs via code generation to tabular data automatic feature engineering clever and clean - Having interpretable features is a big plus and an advantage over prior automatic methods - If I'm not mistaken, the first positive showing of LLMs in tabular data (where using an LLM seem to improve the performance of an already strong tabular model) - Percussion taken to create a non-leaked test-set

Weaknesses

The major issue (that's why the score is not as high as it could have been for me) is the lacking evaluation. Both in terms of baselines/complementary methods and datasets. - Authors compare and complement the proposed method with AutoFeat `[1]` and DFS `[2]`, while much more effective automatic feature engineering methods like FETCH `[3]` and OpenFE `[4]` exist. - Also, in the above-mentioned papers, the evaluations are much more extensive, including more datasets (also regression datasets) and base models. Implementing CAAFE in those benchmarks would be a very big plus for the validation of the proposed method.

Questions

- What do the bold entries mean in the table? The standard deviation values are pretty large, but results are bolded. Were stat tests used? - Does performance saturate after 10 iterations? What is the reason for stopping there? - Do you update the prompt with the generated feature descriptions for new iterations (I think info is missing in the paper, but this seems important)? - Could you report some numerical quantification of features generated by CAAFE, like feature importance ranking? - Minor issues: - line 159: should probably cite TabPFN - line 195: broken reference

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

- I think the fact that method requires dataset and feature descriptions is a limitation - Looking at the generated code, some features and their usefulness explanations are, almost surely, hallucinations. I think It's important to mention hallucinations in the limitations section to represent this. **References** - `[1]` Franziska Horn, Robert Pack, and Michael Rieger. The autofeat python library for automatic feature engineering and selection. - `[2]` James Max Kanter and Kalyan Veeramachaneni. Deep feature synthesis: Towards automating data science endeavors - `[3]` Li, Liyao, et al. "Learning a Data-Driven Policy Network for Pre-Training Automated Feature Engineering." - `[4]` Zhang, Tianping, et al. "OpenFE: Automated Feature Generation beyond Expert-level Performance."

Summary

The paper proposed Context-Aware Automated Feature Engineering (CAAFE) approach to integrate new features learned from dataset descriptions using large language models into AutoML process for tabular datasets. The proposed approach was evaluated using 14 datasets.

Strengths

The paper is well-written with clear research motivation, experiment design and results illustration. It's an interesting attempt to try to integrate large language models into AutoML process. Code scripts are provided.

Weaknesses

1. Overall the use case of adding features from data description is trivial, even using large language models or human-in-the-loop concept. The proposed feature engineering process through prompt generation from the data description or any semantic information looks more of heuristic rules in featurization, which could be performed without large language models. LLM plays the role of automated code generation, instead of machine learning feature engineering approaches (e.g. PCA) incorporated in most AutoML systems. 2. The paper fails to catch up with the latest trends of AutoML systems. Most of existing AutoML systems will perform automated feature engineering, e.g. H2O AutoML, TPOT, AutoGluon, AutoSklearn, etc. Missing those AutoML baselines significantly eliminate the value of the proposed approach in modeling effectiveness. 3. In Table 1, the increased CAAFE performance is relatively trivial for most datasets, except the tic-tac-toe, which significantly impact the mean ROC AUC.

Questions

Highly recommend the author to run the data experiments using the existing AutoML systems (listed in weakness part, they are all open source) without LLM featurizations from dataset description as baselines.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

Yes. The author addressed the limitations.

Thank you...

Thank you for addressing reviewer comments in depths. Given the additional evaluation, I will adjust my overall rating to 7.

Re: Rebuttal by Authors

The elaborate rebuttal is very much appreciated and my concerns regarding CAAFE were lowered by the authors' response and additional experiments. However, I, unfortunately, cannot see the ethics reviewer's comments and thus cannot judge to what extent these issues have been resolved. Yet, it is relatively hard to judge based on the aggregated means without any standard deviation whether the drop in performance is indeed substantial and outside the regime of the standard error. Furthermore, for GPT-3.5 it seems as if it cannot really leverage the semantic information of the feature descriptions at all? How can this be explained?