Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning

Summary

The paper introduces OCTree, a framework that uses LLMs to generate and refine feature generation rules for tabular data. By incorporating decision tree reasoning, OCTree iteratively improves feature generation using feedback from past experiments. The framework enhances the performance of some prediction models, including decision trees and neural networks, and demonstrates improvements on real-world datasets. OCTree works with datasets with or without language descriptions and outperforms existing feature engineering methods, showing flexibility in feature generation for prediction tasks.

Strengths

The paper combines LLMs with decision tree reasoning to generate features for tabular data, offering a novel approach to feature generation. The paper demonstrates improvements through several experiments, and its flexible framework shows potential benefits in enhancing model performance. The paper is well-structured and clearly written.

Weaknesses

1. The framework is costly and time-consuming due to the need for training new models for validation during each iteration, and it involves some manual operations, preventing full automation. 2. Lack of comparisons of training times makes it difficult to assess the efficiency relative to other approaches. 3. The experiments are limited to datasets with a maximum of 54 features, lacking evidence of effectiveness on high-dimensional data (e.g., datasets with hundreds of features). 4. Datasets are sampled to sizes smaller than 50,000 instances, raising concerns about effectiveness and scalability on larger datasets.

Questions

1. Are there other works that use LLMs for feature generation? If so, how does this approach differ from them? 2. What is the time cost associated with OCTree? How much slower is it compared to directly training XGBoost? Does the time cost increase significantly with the size of the dataset and the feature dimensionality?

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

N/A

Summary

The authors propose an automatic feature engineering method called OCTree. The OCTree algorithm uses LLMs to generate new features. The feature is then used in the training of the black box, and the new validation score is stored. The algorithm uses a decision tree to learn rules that show the reasoning behind selecting these new features and incorporate this in an iterative process. NOTE: If the authors address the issues I have raised in limitations and answer my questions, I will improve my score.

Strengths

The paper's research question, automatic feature engineering, is an important area of machine learning research that has not gained enough attention.

Weaknesses

It is very difficult for me to follow the notation and understand what the algorithm is doing. This gets harder as the authors have not released the code for the experiments. The authors keep iterating that the main limitation of another feature engineering is that they require setting the manual search space without showing how their approach is solving this limitation. The experiments do not include CAAFE, the most relevant algorithm to OCTree. The authors claim that the reason behind this is that CAAFE relies on a language-based context of the dataset, making it inapplicable to the datasets they have used. But I am not yet convinced. None of the decision-tree reasoning and how it is improving OCTree is explained in the paper.

Questions

Why have the authors excluded CAAFE? I looked at the tutorials, and they seem very easy to use. Why do the authors say that CAAFE is orthogonal to their approach? I do not think that is the case. I understand that your approach is CAAFE + a decision tree process embedded. Can you elaborate on the difference between your approach and CAAFE with a concrete example? Your notations are so confusing. In Line 131, what does D + r mean? D is a dataset, and r is a rule; they cannot be summed up. Can authors present evidence on how the inclusion of rules learned by decision trees is increasing the efficiency of their algorithm?

Rating

3

Confidence

3

Soundness

2

Presentation

1

Contribution

1

Limitations

The main competitive algorithm in this work, CAAFE, is excluded from the study. Therefore, we do not have a good baseline for how this approach can improve the problem. The authors have not addressed the study's main concern: the risk of memorization. LLMs are possibly trained on the open datasets they are using. This could be that the LLM is, for example, summarizing what most Kaggle blogs and Medium posts on these datasets have done.

Summary

This paper proposes a new tabular learning framework called OCTree (Optimizing Column Feature Generator with Decision Tree Reasoning). The framework leverages the reasoning capabilities of large language models (LLMs) to automatically generate new column features based on feedback from decision trees. Experimental results demonstrate that this framework consistently outperforms existing automatic feature engineering methods across various tabular data prediction tasks. The OCTree framework effectively utilizes the reasoning capabilities of LLMs and feedback from decision trees to automatically generate new column features, significantly enhancing the performance of tabular data prediction tasks. This framework demonstrates the great potential of LLMs in automatic feature engineering for tabular data.

Strengths

S1: A new framework for automatic feature generation that leverages LLMs' language understanding and reasoning capabilities, using feedback from decision trees. S2: All experiments on various real-world datasets show significant performance improvements in different prediction models, including gradient-boosted decision trees and deep neural networks. S3: The method works for both datasets with and without language descriptions and shows good transferability of generated features across different types of models.

Weaknesses

w1: I thoroughly enjoyed reading this work. The authors present a unique perspective on the optimization of tabular data tasks. For example, in the abstract, the authors highlight the shortcomings of existing work, such as "they often rely solely on validation scores to select good features, neglecting valuable feedback from past experiments that could inform the planning of future experiments." This indeed reflects the current issues faced by Automated Feature Transformation, where high-performing features may not necessarily be meaningful features. However, conversely, it seems that the goal of Feature Engineering is to improve the performance of downstream machine learning tasks, which is a very direct objective. Isn't it contrary to the original intent of feature engineering if validation scores are not used as an evaluation metric? w2: The writing of the entire paper is relatively easy to read, although some concepts were not discussed in depth. For instance, the authors only claim that validation scores are a flawed feedback mechanism but do not explore some of the newer methods in the main text, such as the latest approaches utilizing reinforcement learning [1-2] or generative artificial intelligence [3-4]. These methods employ the performance of models like decision trees and other machine learning models as scores to learn strategies. w3: In Figure 1, Fever, Fatigue, Breathing, etc., are used through processes like Tabular Data Understanding and rule generation to combine boolean forms (Fever, Fatigue) or one-hot forms of Breathing to create a new feature, Smoke. This is quite intuitive; however, if the rules generated in step 1 are inaccurate, the generated column might also be inaccurate. Moreover, can the concept of whether one smokes or not be automatically and adaptively formed during the generation of decision trees or the training of neural networks, rather than requiring such an inaccurate combination? If not, do the authors have experiments or specific mathematical proofs to validate this claim or to show that neural networks, for instance, would converge more difficultly without these generated key features? w4: In the Column name generation section of the Methodology (section 3.2), how do the authors ensure that the generated column names are valid? w5: In Table 2, it seems that some datasets with non-linguistic descriptions did not improve or only showed general improvement. What are the specific reasons for this? Although this work seems overly intuitive at times, I remain excited about the application of large language models in this field. I hope the authors can address the above questions. [1] Traceable group-wise self-optimizing feature transformation learning: A dual optimization perspective, TKDD [2] Group-wise Reinforcement Feature Generation for Optimal and Explainable Representation Space Reconstruction, KDD [3] Reinforcement-Enhanced Autoregressive Feature Transformation: Gradient-steered Search in Continuous Space for Postfix Expressions, NeurIPS [4] DIFER: Differentiable Automated Feature Engineering, ICAML

Questions

see weakness

Rating

6

Confidence

5

Soundness

2

Presentation

3

Contribution

3

Limitations

see weakness

Summary

The authors introduce a novel feature engineering technique that leverages LLMs for language-based reasoning and considers the outcomes of past experiments as feedback for iterative rule improvements.

Strengths

- the authors introduce a novel feature engineering technique that leverages LLMs so that past experience can be considered when proposing novel features. - the paper is well-structured, the results are clearly presented, and multiple LLMs are assessed

Weaknesses

- the authors did not compare their method against other techniques for automated feature generation - the authors provide no details on whether several rounds of LLM prompting were executed, how do results vary across multiple executions and whether they had issues with LLM hallucinations

Questions

We consider the paper interesting and relevant. Nevertheless, we would like to point to the following improvement opportunities: GENERAL COMMENTS (1) - The authors compare their model against models on which such feature generation was not applied, showing promising results. Nevertheless, we miss a comparison against similar methods that generate features either with heuristics or LLMs, as explained in the Related Work section. In particular, we would encourage the authors to include some comparison against the following work: Hollmann, Noah, Samuel Müller, and Frank Hutter. "Large language models for automated data science: Introducing caafe for context-aware automated feature engineering." Advances in Neural Information Processing Systems 36 (2024). (2) - Did the authors execute several rounds of feature generation using the same prompt and input? Did they observe any variability regarding the results obtained? (3) - How are results different when considering different LLMs? Are there certain patterns regarding the features generated/not generated? Furthermore, the authors mention that "incorporating code datasets alongside dialogue datasets may further enhance the performance of the rule generator LLM". How were such prompt results better from regular LLMs and different from those generated by the LLM proposed by the authors (e.g., in Table 3)? (4) - Did the authors face LLM hallucinations? We would appreciate a brief description on how did they handle such cases and some quantification, to understand how frequent such problem is across different LLMs they tried. FIGURES (6) - Figure 3: (i) can be made monochromatic, (ii) are datasets balanced? How is the cut threshold determined to measure Accuracy? Could this be replaced with a threshold-independent metric? TABLES (7) - Table 7: bold the best results

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors acknowledge some limitations of their work. Nevertheless, we consider some additional weaknesses (see weaknesses section) should be addressed and eventually acknowledged.