Symbolic Regression with a Learned Concept Library

Summary

This paper proposes a way to incorporate LLM prompting to improve symbolic regression. They uses PySR, a standard symbolic regression library, as their base SR algorithm. Then they add LLM prompts to different SR algorithm steps: population mutation, crossover, and initialization. They replace the PySR implementation with LLM prompted implementations 1% of the time. The LLM prompts are based on identifying high level concepts and prompting the LLM to do its operation using the concept as a suggestion to follow. High level concepts are tracked using an abstraction prompt given high performing expressions, and are evolved using LLM prompting as well. The authors show that augmenting PySR with LLM concept-based prompting solves around 7/100 addition tasks from the AI Feynman SR benchmark, and show that PySR + LLM (LaSR) performs better on a synthetic task designed to test for data leakage.

Strengths

- Well written, great figures - The framework for integrating LLM-based SR into PySR is well-designed, and could in principle work for other prompting approaches besides the concept-based SR. The LLM-mutation, crossover, and initialization steps could be replaced with other LLM based techniques. Cool! - Using LLM's to learn concepts to guide SR is a well-motivated choice of prompting technique, given the importance of high level concepts for human equation discovery - Based on the literature review comparing LaSR to two other LLM SR tools, LaSR seems like an original contribution. - Given existing literature on program synthesis with library learning, LaSR is a great approach that bridges the gap a bit between SR and program synthesis, as done with modern tools. - LaSR is also a good contribution to the growing body of work on library learning and its benefits for search. It is very similar to LiLO, which is built off DreamCoder, but applied to symbolic regression. - The authors have a strong analysis of LLM incorporation based on (1) algorithmic cost in terms of millions of tokens per iteration, and (2) comparing GPT 3.5 with open source llama 8b on the results.

Weaknesses

- It's not clear how valuable the concept abstraction approach is compared to some "baseline" of simple LLM prompting. For example, one baseline could just use a single concept "This expression is a good formula for symbolic regression" or something like that, and see how it compares. This could perhaps be a direction for future work: try a bunch of simple prompting strategies for combining LLM's with PySR, and report how well each of them work. - I'm not sure what to make of the synthetic dataset. In particular, PySR works so well on AI Feynman alone, but works very poorly without the LLM addition on this synthetic dataset. Why does LaSR work so much better than PySR here, but not help as much on AI Feynman? One hypothesis is that AI Feynman has a lot of easy tasks, and the synthetic benchmark only has tasks right on the edge of what PySR can discover, which LLM incorporation helps push over the edge. Another pessimistic take is that the synthetic benchmark is designed with LaSR in mind. While this still eliminates worries on data leakage, an explanation here would help understand better how LaSR is being helpful. I'd like to emphasize that including answers to these questions (both of which could suggest negative results) in the paper would not decrease my review score.

Questions

See weaknesses section. Some other suggestions: - fix backwards quotations marks

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

N/A

Summary

This paper introduces a method that learns a library of concepts (natural language description of helpful and unhelpful concepts) as a means of guiding genetic search for symbolic regression. The core idea is that such concepts can be used to bias genetic operations through an LLM. The method was evaluated on the 100 Feynman equations and on a synthetic dataset. The paper also includes ablation studies on the various components of the system.

Strengths

The paper presents a creative form of using LLM to speed up the search of genetic algorithms for symbolic regression. Instead of simply storing a library of programs, as GA algorithms do, the algorithm also stores a library of natural language concepts. Such concepts can be seen as abstractions of the population of programs encountered in search. Given in natural language, the abstractions can be used to drive the search as an LLM can be used to generate programs based on the description. Such a creative approach! The idea presented in this paper is general and can be more broadly applied to other problems. I can already see how I could use a similar approach in my own research! Another strength of the approach is the author's care with data contamination. Initially, I was skeptical of the approach as it uses LLMs to solve problems whose solutions are available online. The authors then explain that the way the LLM is used is unlikely to allow it to simply retrieve the solution from its training data. The explanation makes perfect sense since the LLM is used to extract concepts from programs the GA generates, and they can't encode the solution available online. In addition to this explanation, the authors also included an experiment on synthetic data showing the advantages of the learned concepts over the search alone. Nicely done! I also enjoyed the fact that the system is built on top of PySR, which is a very efficient system for symbolic regression. This eliminates the possibility that all the gains the LLM provides could be easily washed with clever engineering. The current results already show that clever engineering alone is outperformed by the system.

Weaknesses

The paper also has a few weaknesses. **Claims that need to be fixed** Some of the claims in the paper are a bit strong and I suggest toning them down. While the leakage explanation the authors provided is reasonable, I would be careful in claiming state-of-the-art performance. When writing "LASR achieves a higher exact solve rate than all other baselines," it is worth mentioning the possibility of leakage. Another claim that seems to be incorrect is the following: "LaSR's increasing the backbone model size and the mixture probability significantly enhances." I think the authors meant to write "substantially" and not "significantly" as there is no statistical test involved. I would also explain why these results are substantially better as the number of problems solved isn't much larger. The explanation I gave to myself is that solving each of these equations is very difficult, so solving one new equation is already quite an achievement. Another claim that needs to be adjusted: "demonstrating that LaSR's performance gains are not rooted in memorized responses." The experiments on the synthetic dataset do not demonstrate this. The experiment with the synthetic dataset is almost independent of the experiment with the 100 Feynman equations. What the experiment demonstrates is that LaSR can outperform PySR even when data leakage is not possible. **Discovering what is known** Perhaps an unfair criticism of the paper is that the method it introduces is used to discover things we already know. I understand that the bar would be way too high and it would be unhealthy to the research area if we required the discovery of new things with the presentation of novel approaches to scientific discovery. So I do not make this criticism as a means of arguing for rejecting the paper (I think the paper should be accepted), but more as a reflection of what the community has been pursuing. The hope is that systems such as PySR and LaSR will eventually be used to make actual discoveries. **Lack of Running Time** I missed the running time of LaSR and PySR in the paper. How do they compare with LaSR making calls to an LLM?

Questions

1. In line 83, shouldn't it be $P(C)$ instead of $P_C$? 2. In the mutation operator, it is stated that for each deleted subtree, one adds a single leaf node (lines 111-112). Is this correct? Why not add an entire new subtree? 3. In the for-loop in line 4 for Algorithm 1, shouldn't it be $I$ instead of $N$? 4. Why not provide a figure in the appendix to describe Concept Evolution (similar to figures 4 and 5)? 5. How does that cascade work for PySR? (the one that didn't work) 6. Why use MSE and not the exact correct metric in Section 4.3? 7. Why is the difference between PySR and LaSR so large in Table 3, but somewhat small in Table 1? 7. Weren't the sentences in Section 4.4 swapped? To me, the first sentence looks more refined than the second. This is because it mentions "waveforms or periodic phenomena," while the second only talks about "scientific phenomena." Perhaps the argument in that section needs some rethinking? 8. In the prompt shown in Figure 5, how much is the excerpt "relation to physical concepts" contributing to the quality of the concepts?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper lists all limitations I could think of, either in the last section "limitations" or throughout the paper.

Summary

This work focus on symbolic regression. They enhaned the traditional method like genetic algorithms by inducting a library of abstract textual concepts. The algorithm, called LASR, uses zero-shot queries to a large language model to discover and evolve concepts occurring in known high-performing model to discover and evolve concepts occurring in known high-performing hypotheses. The algorithm can be seen as a kind of hybird of evolutionary algorithm and LLMs. Through experiments, LASR substantially outperforms a variety of state-of-the-art SR approaches.

Strengths

1. This work proposed to introduce a concept library in symbolic regression, which is really similar to how human works, so the idea make sense. For the introduction of the library, this work also leverage abstrction or understanding ability of LLMs to design three phrases process, concept evolution, hypothesis evolution and concept abstraction. The design mix the strenghes of evolutionary algorithms and LLMs. 2. The experimental results are good comparing to those other baselines of learning-based or evolutionary-based.

Weaknesses

1. Introdution of LLMs would inevitably raise the cost for the task comparing those traditional algorithms. 2. (this could be a question) The introdution of concept library seems to not making sense in all cases, imaging that we are facing some totally unknown black-box environment, the backbone function could be something random or out of the knowledge we have. In this setting, the traditional evolutionary algorithms might perform better because they do not have this kind of knowledge as the constrant.

Questions

1. What's the operation of LLMINIT? If we do not have any prior knowledge, then what will we prompt LLMs to do the initialization? Is it the same when we nothing more to know in advance. 2. The initilization, mutation and crossover operation in LASR hybrid traditional evolutionary algorithms and LLMs, so what's the intuition to do this? Because there are several works that directly replace genetic operation with LLMs operation, such as (https://arxiv.org/abs/2401.02051). I am curious about have you ever tried to use merely LLMs for the whole process, that would be an interesting topic. 3. Is LASR capable to resolve high-dimentional symbolic regression tasks? Because current research about symbolic regression struggle to due with high-dimensional settings. 4. Have you ever tried to compared the work KAN (https://arxiv.org/abs/2404.19756), which can also be used in symbolic regression and report a great results.

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

See weakness and question parts.

Summary

This paper introduces LASR, a symbolic regression framework that enhances PySR by incorporating Large Language Models (LLMs) to discover and evolve "concepts" from high-performing equations. These concepts are then used to guide the search process. LASR is evaluated on the Feynman equations dataset and a set of synthetic tasks, showing improved performance over existing symbolic regression baselines.

Strengths

The idea of integrating LLMs into symbolic regression to learn and use abstract concepts in terms of natural language is interesting. The methodology is well-structured with clear explanations of the algorithm components.

Weaknesses

My main concerns are regarding the evaluation. Experimental design and analysis are insufficient to convincingly demonstrate the method's advantages over existing approaches; specifically: There are serious concerns about potential data leakage and unfair advantage when using LLMs on well-known, simple physics equations that may be part of LLM training data. While there is a section on the data leakage validation, its limited evaluation scope and lack of comprehensive analysis on more complex and real-world datasets, makes it difficult to assess the true capabilities and generalizability of the method.

Questions

* Could you provide the 7 additional equations that LASR solves beyond PySR as well as the forms found by LASR? Are these 7 equations consistent across runs? What have been specific contributions of the LLM for them? * How can you ensure that the LLM is not implicitly using its prior knowledge to generate simple Feynman physics equations? The ablation study shows that variable names and hints improve performance, which raises concerns. For instance, variable names like "g" for gravity or "Kb" for the Boltzmann constant could trigger the LLM to suggest the well-known relevant equations. * Have you conducted ablation studies that remove all physics-related terms, variable names, and any mention of physics or internal knowledge from the prompts? This would help (to some extent) isolate the LLM's ability to learn purely from data patterns. * For equations solved by both LASR and PySR, could you provide the number of iterations required to obtain the correct form using each method? * Could you clarify what constitutes an "iteration" in your experiments? There are discrepancies between Figure 1 (10^6 iterations), the main experiments (40 iterations), and the synthetic data experiments (400 iterations). Could you explain these differences and provide results for more iterations (maybe around 10^6 iterations), particularly for the synthetic dataset? Additionally, could you clarify how the concept library operations frequency (every 400 iteration based on Figure 2), and LLM usage frequency (p=1%) translate to these small iteration counts? For example, for 40 iterations, how many time LLM operations and concept library operations will be used? * For the data leakage validation, why were custom synthetic equations used instead of existing datasets like black-box datasets in SRBench? This would provide a more standardized comparison and better demonstrate LASR's capabilities on real-world problems. * Could you provide a computation time comparison between LASR and PySR for experiments with more iterations, to assess the computational trade-offs of using LLMs? * Could you provide examples of "hints" that are provided in experiments of Feynman equations? * Based on Table 2, increasing the value of p seems to improve performance. Have you considered analyzing higher values of p, especially for GPT-3.5? Could you explain why higher values were not explored?

Rating

4

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

See above.

Slight corrections.

Due to a formatting error, it seems part of our response to your questions ended up in our response to Reviewer CCz9. We copy the relevant portions verbatim in this comment: > **Discovering what is known…** Our experiments on synthetic data suggest that we can use LaSR to make discoveries even without significant prior domain knowledge. In addition, since the submission, we have been actively exploring ways to use LaSR in novel discovery tasks. For example, we currently have exciting preliminary results on automatically discovering novel LLM scaling laws using LaSR. Such real-world case studies require deep collaboration with domain experts and are hence outside the scope of the present foundational effort. However, we anticipate many such results in the future. > **Lack of Running Time** Our goal is to present experimental results that can be replicated reliably by the broader community. As such, we root our experiments in the number of iterations instead of wall time. This is because the wall time is heavily dependent on the LLM backend, the number of tokens each query uses, and the network quality. For instance, for `gpt3.5` we cannot reliably measure the runtime as the backend is slower during periods of high traffic. Even for local models, vLLM's query scheduling is non-deterministic so the wall time will be substantially different across experiments. Regardless, we were measuring around one second per query using vLLM. Since we make 60k calls per iteration (Appendix A.3.1) the amount of time for p=1% would be around 600 seconds / 10 minutes per iteration. PySR roughly takes 30 seconds per iteration. We expect performance to improve as we increase CPU/GPU parallelism, and as the hardware to run LLM inference improves. `[1]`: https://github.com/stanfordnlp/dspy

Response to Authors

Thanks for your response! All of my concerns have been addressed. Good work and I will maintain my score.

Response to the authors

Thank you for your responses. Some of my question have been answered. But I still have some concerns including: 1. For the lack of experiments on SRBench black-box datasets, the authors mention that they wanted to experiment on datasets with known ground-truth equations. However, first, the authors have not reported on the function recovery for this dataset. More importantly, LaSR is eventually compared based on fitness (R2) in sec 4.5 and table 3, which could have been the case for SRBench black-box datasets. 2. I understand that using LLMs can be time-consuming, and this is in no ways a downside. However, I find comparing LaSR and PySR on same number of iterations (and with only 40 or 400 iterations in Fig.3 and table 3) not fair, since they require very different computation and wall-clock time. Also, since PySR is a GP-based method and lacks prior knowledge, it would be reasonable for them to need more number of iterations to converge to the same level of performance. So it would be more reasonable to report the results after all the methods have converged (or at least same computation/wall-clock time), and not necessarily at the same iteration. 3. In the current results shown in Fig 3, without the additional memorization hints (variable name, hints, or mention of physical concepts), the improvement of LaSR (which in this ablation could be potentially attributed to the LLM role in learning good vs. bad forms) looks marginal. Also, this plot shows the results on MSE and not the exact match as reported in table 1. So it is not shown that how many equation are discovered out of 100 in this condition. Based on figure 3, PySR has solved ~41 equations after 40 iterations on MSE metric but based on table 1, it has solved 59 equations based on exact match. This discrepancy in the metrics and the final results of LaSR in this condition should be explained. Other concerns and comments: As an additional note, the authors mention that in table 1, the 7 improved equations are consistent across all runs. As both initialization of GP methods and the responses of LLMs involve some level of variance, this would increase the concern that these 7 enhanced equations could have in fact obtained due to the memorization rather than pure reasoning and learning on the equation forms. The observation on the form of discovered equations is interesting and worth discussing, however, not generating the exact known form of equations does not guarantee that the model has not used its internal memorization as a part of its inference. I would suggest authors to include a discussion in the paper on the results of LaSR without any variable name and even a notion of "physical concepts".