Reinforcement-Enhanced Autoregressive Feature Transformation: Gradient-steered Search in Continuous Space for Postfix Expressions

Summary

Feature transformation is an effective way to improve downstream task. This paper aims to improve the search efficiency of the optimal feature space while ensuring the stability and robustness of the transformation. They formulate the discrete Automated Feature Transformation (AFT) problem as a continuous optimization task and propose a reinforcement-enhanced autoregressive feature Transformation framework (MOAT). MOAT implements four steps to advance efficiency and robustness. Extensive experiments and case studies are performed to demonstrate the effectiveness and robustness of the proposed method.

Strengths

This paper provides a clear and detailed illustration of the proposed framework, especially the modules in the framework. They conduct extensive experiments to demonstrate the superiority of the framework, including ablation studies for each module, to demonstrate the efficiency and robustness of the framework.

Weaknesses

Missing reference to experimental results on page 8, Data collection and augmentation section. Several methods have used reinforcement learning to improve search efficiency at present (e.g., in data augmentation, NAS), and the framework proposed in this paper lacks novelty to some extent.

Questions

(1) MOAT uses multi-stage process to obtain a convincing performance. Is it possible to learn an optimal feature space using reinforcement learning by end-to-end? (2) Does the optimal feature space searching problem can be directly solved by representation learning in continuous space, rather than combining multiple operations in discrete space？

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

This paper pays more attention to framework design and lacks some theoretical perspectives. In addition, how to design the framework proposed in this paper into an end-to-end training process is desirable.

Summary

This paper introduces a succinct yet effective framework for automatic feature transformation. The authors mapped decision sequences collected from reinforcement learning into a high-dimensional vector space. Optimization is carried out through the gradient direction provided by an evaluator, and a sequence reconstruction component is employed to rebuild the decision sequence of this feature transformation. Overall, the paper's experiments are comprehensive, and the discussions provide excellent validation of various features of the framework. The comparisons with existing models also aptly place this work among the latest contributions in the field.

Strengths

1. The design of the entire framework is clear and comprehensible, and based on the authors' descriptions and experimental discussions, this framework also shows potential for reuse in other fields. 2. The experimental design is detailed and particularly addresses my concerns about the time-consuming nature of using reinforcement learning for data collection. 3. Similar to the second strength described above, a significant issue with using reinforcement learning in feature transformation tasks is that it might require an excessive number of search steps to achieve optimal. However, this framework can use a fixed number of search steps to construct the space for searching and achieve respectable downstream task performance, significantly mitigating the problem of uncertain duration in feature transformation tasks based on reinforcement learning.

Weaknesses

1. There are a few typos in this paper, e.g., the legend of the figure 4, the model name is wrong. 2. The authors have proposed an encoder-decoder model based on LSTM for sequence modeling. However, there is no discussion on why this specific method was chosen. It would be beneficial if the authors could explain the rationale behind this choice. Could other sequence modeling methods, such as Transformers, have been considered or used instead? 3. While the design of the model and ample experiments mutually validate the feasibility of this framework, if there could be more in-depth insights provided, it might help establish this work as foundational for broader fields.

Questions

1. The framework's description of the data collection part for the optimization objective lacks clarity. The authors should consider adding more detail in the appendix to better elucidate this aspect. 2. Are there any way to make the framework more generalized, e.g., extend this method to graph-like data, picuture-like data. 3. Would it be feasible to design a more generalized operation set that views these mathematical operations as analogous to sub-networks?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

None

Summary

The authors propose a feature transformation method that reformulates the problem as a continuous space optimization task and utilizes a reinforcement-enhanced autoregressive framework for gradient-steered search. The method involves four steps: (1) reinforcement-enhanced data preparation, (2) feature transformation operation sequence embedding, (3) gradient-steered optimal embedding search, and (4) transformation operation sequence reconstruction. The proposed method is evaluated through extensive experiments and case studies, demonstrating its effectiveness and robustness.

Strengths

1. This paper proposes a novel automated feature transformation framework that has not been explored before, and it achieves significant improvement in performance compared to previous methods. 2. This paper provides clear and detailed explanations of the proposed method's four-step process, making it easy for readers to follow and replicate the method. 3. This paper conducts extensive experiments and case studies to illustrate the effectiveness of the framework from different perspectives. 4. The authors released the related code and data, which can help other researchers reproduce the experiments. 5. This paper provides clear and detailed explanations of the proposed method's four-step process, making it easy for readers to follow and replicate the method.

Weaknesses

1. The authors use LSTM as the backbone of their framework. Any reason for choosing LSTM? How about other alternatives, such as Transformer? 2. There are some typos in this paper. For instance, in Figure 4 (c) and Figure 4 (d), the name of the model variant should be MOAT^-a instead of GBFG^-a. The authors should fix these typos for keeping consistency. 3. Analyzing the experimental results, the RL-based data collector is important. But, the description for this part is limited. Can the authors provide more explanation on this?

Questions

1. It is unclear why the performance of DIFER is not good for openml 616, openml 637. Can you provide some explanation on this? 2. After reviewing the appendix section, I found that MOAT requires more training time but less inference time than DIFER. Can the authors explain the reason for it?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

None

Summary

Distinct from existing work, this study collects feature transformation operation sequences, which have a well-researched background based on reinforcement learning. It then obtains hidden representations of these limited numbers of sequences in a self-supervised manner and finally optimizes this continuous vector representation to generate superior sequences guided by gradients. Specifically, to model the sequences, the authors propose an LSTM-based encoder-decoder-evaluator architecture with just a few collected samples. Detailed runtime analysis was conducted in this work to demonstrate its advantages over reinforcement learning-based (GRFG) and random generation-based (DIFER) approaches. The experiments, coupled with the authors' analytical discussions, effectively substantiate the viewpoints declared in the paper.

Strengths

1) This paper is well-written and easy to understand, with an appropriate level of detail in describing the methods. It considers the primary issue, time consumption, associated with this framework in terms of experimental design, effectively addressing the readers' concerns. 2) The paper's model architecture is thoughtfully designed, and the authors articulate their motivations for each component with great clarity. The streamlined nature of the entire framework demonstrates the depth of the authors' understanding and thoughtful consideration of this work. 3) The authors provide thorough experiments and analysis for this work, covering aspects of interest in this field such as performance, robustness, runtime, scalability, memory usage, traceability, and efficacy tests for each component. The visualization of the hidden space well illustrates the reasons for the model's effectiveness in conducting gradient search.

Weaknesses

1) In the sections 2.1 Important Definitions and Section 3.2 Reinforcement Training Data Preparation, the authors have inconsistencies in the naming of the agents within the cascading agent structure for the data collection part. For instance, it's named as 'feature agent1' in the definition section, while it's referred to as 'head feature agent' in the methodology part. The authors should maintain consistency in these definition names to reduce confusion for the readers. 2) The description of the optimization objective for reinforcement learning in the data collection section is not detailed enough. The authors should provide a more comprehensive description of this critical component.

Questions

1) Why does DIFER require more time in the inference phase? The authors should provide a detailed explanation for this. 2) In the time complexity analysis, the authors have chosen four datasets - Wine Red, Wine White, Openml_618, and OpenML_589. However, they have not adequately described the rationale behind selecting these particular datasets. 3) There are some avoidable writing errors, such as on page 13 of the appendix where the authors incorrectly referred to Table 3 as Figure 3. The authors should conduct a thorough check of the entire paper to prevent inconsistencies in naming.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

None