Transformer-based Planning for Symbolic Regression

Summary

The paper introduces TPSR, a Transformer-based Planning strategy for Symbolic Regression. TPSR incorporates Monte Carlo Tree Search into the transformer decoding process, enabling the integration of non-differentiable feedback such as accuracy and complexity. Experimental results show that TPSR outperforms existing methods in terms of fitting-complexity trade-off, extrapolation abilities, and robustness to noise.

Strengths

- The paper is well written and easy to understand. - The idea of enhancing large scale pre-trained Transformers with improved search capablities is very promising in the context of symbolic regression - The model shows good performance both compared to the E2E baseline and the GP methods.

Weaknesses

- My main concern is about the novelty of the approach. A very similar idea has been recently investigated in [1] where the authors also combine MCTS with pre-trained Transformers. I would be grateful if the authors could clarify any eventual differences between the two approaches. - The impact of $\lambda$ seems quite significant in your experiements. However, it is not clear to me how one should select it in practice. [1] Kamienny, Pierre-Alexandre, Guillaume Lample, and Marco Virgolin. "Deep Generative Symbolic Regression with Monte-Carlo-Tree-Search." (2023).

Questions

Please refer to the weakness part above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

n/a

Summary

Authors propose a transformer-based planning (using MCTS) strategy to solve symbol regression task. Different from traditional decoding method, the new method is able to integrate non-differentiable feedback into the transformer-based process of equation generation. Experiments demonstrate the significent performance.

Strengths

Distilling symbolic equation from noisy data is intractable. Recent progress is achieved by training neural networks to generate candidate symbolic expressions, which is really promising. This work combines the Monte Carlo Tree Search and pretrained transformer-based symbol regression model for equation generation. Compared with Genetic programming method, the new approach not only leverages pre-trained priors, but also considers feedbacks during the generation process.

Weaknesses

There is not much initiality in the new method. It demonstrates a new application for a combination of two existing methods.

Questions

Are there experiments to show changes of performances, if we change the selection set of mathematical operators and symbols? Can out-of distribution data be identified, and used for the promotion of the symbolic regression process? It might not be diffiuclt for symbol regression method to find laws, such as f=ma. Could it find E = m c^2?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Monte Carlo Tree Search is statistical, and Pre-trained transformer is trained through data, the integration of the two methods is still within the traditional paradigm of machine learning, so, may not work well for out-of distribution data.

Summary

This paper proposes to incorporate Monte Carlo Tree Search (MCTS) on top of pretrained transformer-based SR models to guide equation sequence generation. This is to address the challenges where existing methods purely rely on the pretrained transformer’s output and without accounting for external performance requirement. In MCTS, the authors develop a reward function to encourage the balance between fitting accuracy and regulating complexity for the SR generation. Also, the caching tricks are employed to improve the implementation efficiency. SR benchmark datasets are used to demonstrate the improved performance of the proposed method over the state-of-the-art.

Strengths

Including performance feedback in the pipeline of generation of SR equation generation from pre-trained transformer-based SR models is well-motivated. To achieve so, this paper proposes including MCTS as the decoder in this pipeline, and imparting the external requirement, via a reward function in MCTS, to eventually improve the performance of equation generation. The extensive experiments and baseline comparison clearly show the effectiveness of the proposed method, in terms of fitting-complexity trade-off, extrapolation abilities, and robustness to noise. The presentation of techniques is clear, and the evaluation in my opinion is solid. Overall, this paper makes a good contribution in the SR field.

Weaknesses

I only have two comments: - In Equ. (1), how to select $\beta(s)$? It would be better to show its effect on the performance in the ablation study as well. - Currently, the method still relies on a pre-trained transformer SR model. The authors could give some perspective about how (or if it is possible) the MCTS can also be incorporated in the transformer training (or fine-tuning) process. -A typo in line 151: trnasformer--> transformer

Questions

See my comments in the above section.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

See my comments in the above section.

Summary

This submission proposes a neural network-based approach to symbolic regression (SR), namely generating equations as sequences. It leverages the power of pretrained SR transformer models and the MCTS algorithm to tradeoff the fitting accuracy and equation complexity. Experimental results on the SRBench and the In-domain Synthetic datasets demonstrate that the proposed approach outperforms the backbone E2E transformer model.

Strengths

Soundness: The techniques employed in the proposed approach are sound. The approach is able to use any non-differentiable target function to guide the training of a neural model for symbolic regression. Experiments demonstrate that it outperforms a state-of-the-art transformer model which is used as the backbone in the proposed approach, indicating that the implementation of the proposed approach is likely to be correct. Presentation: The submission is in general well written and organized, easy to follow.

Weaknesses

Presentation: There is a minor issue on the term single-instance symbolic regression introduced in Related Work. According to the description about the therein algorithms GP, RL, GP+RL and MCTS, the difference between them and the proposed approach mainly lies in not employing pretrained knowledge. Thus, the term single-instance is strange and cannot tell the true difference from the proposed approach. Contribution: The proposed approach seems to be a combination of the E2E transformer model [18] and the MCTS framework [26]. Although the transformer model can be replaced with other neural network-based models, the contributions beyond [18] and [26] are not significant. Moreover, the current evaluation cannot confirm that either the proposed approach is a general framework for enhancing any neural network-based model for symbolic regression, or the approach achieves the truly state-of-the-art performance. For the former confirmation, the authors need to compare multiple implementations having different backbone models with the original backbone models. For the latter confirmation, the authors need to compare the proposed approach with more state-of-the-art solutions such as [30] and [31].

Questions

Why the algorithms GP, RL, GP+RL and MCTS used for symbolic regression are called single-instance?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

As far as I can see, the authors have adequately addressed the limitations through sufficient discussions in the supplemented material.

Summary

This paper proposed to combine pretrained Symbolic Regression models with MCTS procedure to improve SR performance without finetuning the pretrained models. Experiments are conducted to demonstrate the improved performance the proposal.

Strengths

1. Proposed a new MCTS based decoding procedure to improve pretrained SR models performance without finetuning the models. 2. The paper is clearly written and easy to follow in texts. The experiments clearly demonstrates the performance improving over the baselines and the E2E models' decoding.

Weaknesses

1. A few key building blocks needs to be summarized from the literature to be a self-contained paper, e.g. how the datasets are embedded. 2. The methodology contribution is minor as only MCTS is introduced on top of SR models although experimental performance improvement is observed. Although this is a valuable contribution it might not meet the bar for NEURIPS.

Questions

1. Please provide more statistics of equation (1)'s terms. For example, would N(s) be mostly 1s in your setting? Please be more specific about what "visit count" means exactly. Will the sub-routine beam search generated sequence be accounted as a visit? Will cache hit be counted as visit during beam search sub-routine? 2. Figure 7, please clarify how you count the number of generated candidates. Are the sub-routine beam search generated equations are counted? Will this difference render the results differently? Minors: 1. In the abstract, "GP-based methods" should use the full name of GP for broader readers' convenience. 2. Line 48, there should be an indent space at the beginning of the sentence. 3. In section 3, to be self-contained, please succinctly re-iterate the key components of the underlying SR pretrained models, e.g. dataset embedding. 4. Line 188, please briefly explain why this is still a Q function in standard MDP framework. 5. Section 4, please be specific whether E2E and TPSR are using exactly the same experiment settings except the MCTS and beam search difference. If possible, a table in the supplemental materials comparing the experiment settings across different approaches might be helpful. 6. Line 244, in the equation of R^2, is \bar{y} the average values of y in N_{test}? Please clarify.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

NA