Library

Efficient Symbolic Policy Learning with Differentiable Symbolic Expression

Deep reinforcement learning (DRL) has led to a wide range of advances in sequential decision-making tasks. However, the complexity of neural network policies makes it difficult to understand and deploy with limited computational resources. Currently, employing compact symbolic expressions as symbolic policies is a promising strategy to obtain simple and interpretable policies. Previous symbolic policy methods usually involve complex training processes and pre-trained neural network policies, which are inefficient and limit the application of symbolic policies. In this paper, we propose an efficient gradient-based learning method named Efficient Symbolic Policy Learning (ESPL) that learns the symbolic policy from scratch in an end-to-end way. We introduce a symbolic network as the search space and employ a path selector to find the compact symbolic policy. By doing so we represent the policy with a differentiable symbolic expression and train it in an off-policy manner which further improves the efficiency. In addition, in contrast with previous symbolic policies which only work in single-task RL because of complexity, we expand ESPL on meta-RL to generate symbolic policies for unseen tasks. Experimentally, we show that our approach generates symbolic policies with higher performance and greatly improves data efficiency for single-task RL. In meta-RL, we demonstrate that compared with neural network policies the proposed symbolic policy achieves higher performance and efficiency and shows the potential to be interpretable.

cs.LGcs.AIReinforcement Learning in RoboticsMachine Learning and Data ClassificationEvolutionary Algorithms and ApplicationsComputer Science
Source

Paper

References (61)

01Learning to Synthesize Programs as Interpretable and Generalizable PoliciesDweep Trivedi, Jesse Zhang, Shao-Hua Sun et al.2021 · Neural Information Processing Systems · 96 citations In Library
02AdaRL: What, Where, and How to Adapt in Transfer Reinforcement LearningBiwei Huang, Fan Feng, Chaochao Lu et al.2021 · International Conference on Learning Representations · 77 citations In Library
03Recomposing the Reinforcement Learning Building Blocks with HypernetworksShai Keynan, Elad Sarafian, Sarit Kraus2021 · International Conference on Machine Learning · 44 citations In Library
04Effective Sparsification of Neural Networks with Global Sparsity ConstraintXiao Zhou, Weizhong Zhang, Hang Xu et al.2021 · Computer Vision and Pattern Recognition · 80 citations In Library
05Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive LearningHaotian Fu, Hongyao Tang, Jianye Hao et al.2020 · AAAI Conference on Artificial Intelligence · 61 citations In Library
06Discovering Symbolic Models from Deep Learning with Inductive BiasesMiles Cranmer, Alvaro Sanchez-Gonzalez, P. Battaglia et al.2020 · Neural Information Processing Systems · 687 citations In Library
07MetaCURE: Meta Reinforcement Learning with Empowerment-Driven ExplorationJin Zhang, Jianhao Wang, Hao Hu et al.2020 · International Conference on Machine Learning · 34 citations In Library
08Program Guided AgentShao-Hua Sun, Te-Lin Wu, Joseph J. Lim2020 · International Conference on Learning Representations · 65 citations
09PyTorch: An Imperative Style, High-Performance Deep Learning LibraryAdam Paszke, Sam Gross, Francisco Massa et al.2019 · Neural Information Processing Systems · 54k citations In Library
10VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-LearningLuisa M. Zintgraf, Kyriacos Shiarlis, M. Igl et al.2019 · International Conference on Learning Representations · 332 citations In Library
11Meta-Q-LearningRasool Fakoor, P. Chaudhari, Stefano Soatto et al.2019 · International Conference on Learning Representations · 167 citations In Library
12Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradientsBrenden K. Petersen, Mikel Landajuela2019 · International Conference on Learning Representations · 487 citations In Library

Scroll for more · 38 remaining

Similar papers

Symbolic Visual Reinforcement Learning: A Scalable Framework with Object-Level Abstraction and Differentiable Expression SearchWenqing Zheng, S P Sharan, Zhiwen Fan et al.2022 · arXiv (Cornell University) · 16 citations In Library
Symbolic Visual Reinforcement Learning: A Scalable Framework With Object-Level Abstraction and Differentiable Expression SearchWenqing Zheng, S P Sharan, Zhiwen Fan et al.2024 · IEEE Transactions on Pattern Analysis and Machine Intelligence In Library
Sample-Efficient Neurosymbolic Deep Reinforcement LearningC Veronese, Alessandro Farinelli, Daniele Meli2026 In Library
Sample-Efficient Neurosymbolic Deep Reinforcement LearningCeleste Veronese, Alessandro Farinelli, Daniele Meli2026 · Proceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems · 1 citation In Library
Interpretable Reinforcement Learning With Neural Symbolic LogicZhihao Ma, Yuzheng Zhuang, Paul Weng et al.2021 In Library
Mitigating Information Loss in Tree-Based Reinforcement Learning via Direct OptimizationSascha Marton, Tim Grams, Florian Vogt et al.2024 · arXiv (Cornell University) · 6 citations In Library
Discrete-to-Deep Supervised Policy LearningBudi Kurniawan, Peter Vamplew, Michael Papasimeon et al.2020 · arXiv (Cornell University) In Library
Learning Symbolic Rules for Interpretable Deep Reinforcement LearningZhihao Ma, Yuzheng Zhuang, Paul Weng et al.2021 · arXiv (Cornell University) · 17 citations In Library
Interpretable Policy Extraction with Neuro-Symbolic Reinforcement LearningRajdeep Dutta, Qincheng Wang, Ankur Singh et al.2024 · IEEE International Conference on Acoustics, Speech, and Signal Processing · 4 citations In Library
S-REINFORCE: A Neuro-Symbolic Policy Gradient Approach for Interpretable Reinforcement LearningRajdeep Dutta, Qincheng Wang, Ankur Singh et al.2023 · arXiv (Cornell University) · 2 citations In Library

Peer review

Accept (poster)NeurIPS 2023 · via OpenReview
Reviewer 9VjU5/10 · confidence 5/52023-06-08

Summary

# I have reviewed this draft once before. This paper proposes a meta-RL method to generate explainable symbolic policies. ESPL contains a symbolic network in the search space and a path selector to find the compact symbolic policy.

Strengths

I think there exists some novelty in ESPL because it contains symbolic + neural structures. Also, the symbolic policies induced by ESPL seem to be effective. The number of experiments is enough. The visualizations are cool.

Weaknesses

Post rebuttal: I upgrade my score to a borderline accept because of the reasonable rebuttal. Although I originally leaned to accept this paper, I think the suggestions from the previous conference are not incorporated, and the draft is not improved much, so this time, I have to vote for rejection. The major concerns are: (1) The proposed symbolic policy looks messy and it does not improve the interpretability much. Also, there could be many optimal policies. (2) The CatPole and other approaches have guaranteed optimal policy. The theoratcal analysis is missing. (3) The authors should try other discrete environments or complicated environments with rich semantics. I feel sorry for this paper. It was very close to acceptance last time but the AC insisted that the novelties were not enough. I don't think a resubmission without major modifications can get this paper in.

Questions

1. Standard errors should be shown. 2. Table3 is a waste of space.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Not discussed.

Reviewer PhEA6/10 · confidence 3/52023-07-04

Summary

This paper proposes to apply differentiable symbolic regression for policy learning. It shows promising results in multiple RL environments (including good average performance and learned interpretable symbolic policies).

Strengths

This paper shows promising results of differentiable symbolic regression in eight RL environments. The model design is intuitive and is shown to achieve good average performance, learn interpretable symbolic policies, and be less likely to overfit for OOD generalization in the meta-RL setting. The model is also more sample efficient than a previous neural symbolic regression model for policy learning.

Weaknesses

Nevertheless, the model is less novel as similar techniques have been explored in domains such as differentiable interpreters (e.g., Terpret [1] or DiffForth [2]) for program synthesis and Dart [3] for neural architecture search/pruning. There are also concerns regarding the experiments: * Symbolic methods are reported to perform better than neural ones even for single-task RL settings, which is counter-intuitive. There should not be overfitting problems in the naive setting, as far as I understand. Why would symbolic methods outperform neural ones? - I find this in appendix, lines 123-124, which seems relevant but could be unfair: > For all the environments, the proposed ESPL performs 3 independent training runs and selects the single best policy. - I also find that they may report averaged performance over multiple runs, which may not be necessary or standard for the single-task RL setting. * Overall, the descriptions regarding evaluation procedures are vague, e.g., - it's unclear whether the final policies evaluated in Table 2 are symbolic or not; and - how to select policies in Table 1; [1] Gaunt, Alexander L., et al. "Terpret: A probabilistic programming language for program induction." arXiv preprint arXiv:1608.04428 (2016). [2] Bošnjak, Matko, et al. "Programming with a differentiable forth interpreter." International conference on machine learning. PMLR, 2017. [3] Liu, Hanxiao, Karen Simonyan, and Yiming Yang. "Darts: Differentiable architecture search." arXiv preprint arXiv:1806.09055 (2018).

Questions

* Why would the symbolic methods outperform the neural ones in the single-task RL settings? * Are there results in more complex/practical environments, e.g., with larger input and output dimensions or longer expressions?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Reviewer uiuA5/10 · confidence 4/52023-07-06

Summary

The paper "Efficient Symbolic Polmicy Learning via Gradient Descent" proposes a new neural symbolic architecture for agents learned via reinforcement learning. Authors propose an architecture with an alternance of symbolic and linear layers. To finally obtain simple expressions, a probabilistic mask is learned altogether with parameters, enabling the production of sparse archiectures resulting in small expressions. The approach is experimented on classical RL settings, and also on meta-rl ones.

Strengths

- Very interesting architecture - Proposals look innovative - Clever way to obtain sparsity - Convincing experiments about efficiency

Weaknesses

- I cannot understand, if the architecture is new, why authors did not experiment it on classical supervised learning (eg, on classification tasks) before such much more reinforcement learning setting. Is the architecture only new in the field of RL ? Is the main contribution of the paper the archiecture or its application to RL and meta-RL ? - I feel that lacks important details about the used archiectures (for instances q is not defined, nothing is said about the choice of symbolic operators, nor about their numbers n and m, etc.) - Not much results about explanability, which is one of the main goal of the proposal

Questions

- Is the main contribution of the paper the archiecture or its application to RL and meta-RL ? what about applying it on supervised learning tasks ? - How is q defined ? is it a deterministic output from a lstm or a gaussian from a transformer or ...? How is it trained ? - How are chosen the sequence of symbolic operators used in the architecture and what are they in the experiments ? how many (please specify m and n) - From my understanding, since we have m+2n inputs in the symbolic layer and only m+n outputs, the architecture has a with that reduces with layers. is it true ? - What is the impact of lmin in (5) ? Do you have experiments to analyze it ? - The metaRL objective given in the end of the first paragraph of page 6 is difficult to understand: the context is both sampled from the policy and used by the policy. Could you give clearer insights about this strange cycling definition ? Also, I would have expect that there is for each new task an exploration phase that would not been considered in the quantity to maximize : for k episodes no rewards, only collecting experience, such that we are good for the k+1th one in expectation depending on contexts from the k first episodes. Wouldn't it be better to present it like this ?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

.

Reviewer iXeo7/10 · confidence 4/52023-07-07

Summary

The paper proposes ESPL, a method for learning symbolic policies in environments with low-dimensional state spaces. ESPL uses a densely connected neural network structure (like DenseNet), where the activations in each layer are replaced with a hand-picked set of functions, such as multiplication, division, log and and exp. Second, ESPL uses the Gumbel-Softmax trick to learn a masking function with a minimal $L_0$ norm. This function ensures that the method learns compact symbolic policies by masking out redundant pathways in the neural network. ESPL is shown to be 100 to 1000 times more sample-efficient than a prior symbolic learning approach in single-task RL. The authors also propose a version that can be conditioned on task context for meta-RL.

Strengths

1. The proposed method is well motivated and clearly explained. $L_0$ regularization of the parameters of Bernoulli random variables is an elegant approach to learning compact symbolic expressions. 2. In single-task RL, ESPL outperforms a prior symbolic learning approach $DSP^0$ with between 100 and 1000 fewer training samples. 3. A task-conditioned version of ESPL called CSP outperforms prior (non-symbolic) approaches in a meta-learning setup. The authors point out that the symbolic policies are much faster to run than the baseline neural net policies. 4. Source code is included.

Weaknesses

1. The discovered symbolic policies in Table 1 seem to be somewhat more complex than the policies from $DSP^0$. It could be useful to additionally measure the complexity of the discovered policies for the various methods (if such measure exists). 2. It is not very clear how much effort went into hand-picking the activation functions in each layer of the network. Is it necessary to hand-design a different network for each experiment / environment? Minor: * Missing space before `(` and `[`. This is repeated at least 10 times throughout the paper, e.g. lines 33 and 34. * Line 109: repeat what ESPL stands for. * Line 210: mixing up $t$ and $T$.

Questions

1. Is Figure 1 right the exact architecture used in the experiments? Or do you hand-design a different arrangement of activations for each experiment? 2. Is it possible to scale symbolic regression to high-dimensional state spaces?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

The limitations of ESPL are not explicitly addressed. I am especially curious about the possibly of scaling up to high-dimensional (e.g. image) state spaces.

Reviewer 9VjU2023-08-10

More detailed comments, and feedbacks to the authors' rebuttal.

Thanks to the authors for their rebuttals. Here are the extended comments and feedbacks on each point. > 1) The proposed symbolic policy looks messy and it does not improve the interpretability much. Also, there could be many optimal policies. Although this paper claims to improve interpretability, however, the learned representations are still very noisy. I'm not sure whether this claim is true. I see that the authors added human experiments. However, honestly speaking I am not convinced by human experiments conducted in such a small temporal period. Besides, I'm more curious why the proposed approach is better than DSP in efficiency. I think both approaches use reinforcement learning, but the authors report that the proposed approach uses much fewer episodes to learn the policy. I'm not sure where the benefits come from. Do the authors compare DSP with an implementation of https://github.com/brendenpetersen/deep-symbolic-optimization? The authors say "These methods cannot be directly applied to learning symbolic policies." It is clear that their Github page contains applications in symbolic policy learning. I'm not sure whether the statement is true or not. > (2) Previous works show that the CartPole and other environments have guaranteed optimal policies [1]. I think the authors should compare their interpretability with [1]. More theoretical analysis of interpretability is encouraged, as previous papers show some proof of the quality of learned policy in simple environments. A smarter way is to not claim superiority in interpretability if the experiments fail to demonstrate this advantage. I think it is better to provide comparison demos (videos or other formats) instead of human evaluations. However, Table 1 only contains results from ESPL but does not show results from related baselines. It is noteworthy that the DSP paper contains several comparisons in Table 1. > (3) The authors should try other discrete environments or complicated environments with rich semantics. I believe the authors should try on MineCraft [2] or other higher-level environments. These semantic-rich environments require more interpretability. The logic is, we do not need a very interpretable policy in the low-level control tasks, such as PID. However, in higher-level planning tasks, we do need more interpretability. The authors respond that they focus on continuous action space. I think it is better to conduct experiments in discrete cases as well. [1] Verma, Abhinav, et al. "Programmatically interpretable reinforcement learning." International Conference on Machine Learning. PMLR, 2018. [2] Sun, Shao-Hua, Te-Lin Wu, and Joseph J. Lim. "Program guided agent." International Conference on Learning Representations. 2019.

Reviewer 9VjU2023-08-11

More suggestions.

1. If the authors still wish to claim superiority in interpretability, it is better to construct some hard cases that baseline models cannot interpret. For the existing environments, I think the DSP's Table 1 is already fairly simple. 2. The title needs to be changed to reveal the true novelty. The current title seems to claim they are the first to conduct symbolic policy learning via gradient descent. But apparently, this is not the truth.

Authorsrebuttal2023-08-11

Thank you very much for the suggestions!

Thanks very much for your further suggestion! We greatly value your suggestions and are willing to provide clarification and make modifications accordingly. 1. Actually, we **did not claim superiority in interpretability** in this paper. The motivation of this paper is to introduce a more efficient method for learning symbolic policies and eliminate the necessity for pre-trained policy models. 2. We are glad to modify the title. Our approach builds a fully differentiable framework and directly uses gradients to update expressions, whereas the previous approach (DSP) introduced RNNS as a proxy to generate symbolic expressions, and they used gradients to update the proxy instead of the symbolic expression itself. We understand your concern that the previous methodology also used gradients, so the title could be ambiguous. How about changing the title to: “Efficient Symbolic Policy Learning with Differentiable Symbolic Expression” or "Efficient Symbolic Policy Learning by Directly Updating the Expression with Gradient"

Reviewer 9VjU2023-08-11

Thank you for your rebuttal.

I think “Efficient Symbolic Policy Learning with Differentiable Symbolic Expression” is a much better title. Also, the abstract and introduction need to be revised. It is better not to claim the interpretability as a contribution. Otherwise, you can tell the truth that the ESPL has a comparable outcome in terms of interpretability. Anyway, I'll increase my score because of the effort of the authors in rebuttal.

Authorsrebuttal2023-08-11

Thank You!

Thank you very much for your time and effort invested in the discussion! We appreciate your positive feedback and really gain a lot from the discussion. We are glad to reply and discuss if there are further concerns.

Authorsrebuttal2023-08-11

Thanks For Your Replay!

>**Q1.** I see that the authors added human experiments. However, honestly speaking I am not convinced by human experiments conducted in such a small temporal period. Besides, I'm more curious why the proposed approach is better than DSP in efficiency. I think both approaches use reinforcement learning, but the authors report that the proposed approach uses much fewer episodes to learn the policy. I'm not sure where the benefits come from. (1) The Human-study showed that the symbolic policies we learned can achieve close interpretability compared to DSP, and the experiment was conducted before the review comments were received. (2) It's important to note that our approach of **reinforcement learning is completely different from DSP**. DSP regards generating a symbolic policy as a sequential decision-making task and employs reinforcement learning to solve it, with the symbolic operation set as the action space. For the reinforcement learning process in DSP, a single reward (or in other word a single training example) requires interactions with the environment over multiple episodes (an episode may involve hundreds or thousands of interactions steps). In contrast, in our method, the learning of symbolic policies is gradient-based and **doesn't involve constructing any additional sequential decision-making process**. Each step of interaction of the symbolic policy and the environment serves as direct training example. Additionally, we have developed an off-policy training approach, which allows each training example to be used multiple times during training. Furthermore, when dealing with multi-dimensional actions, DSP needs to construct multiple sequential decision tasks. Simultaneously optimizing these tasks is challenging, leading them to use pre-trained neural network policies as anchor models and alternately optimizing each dimension of action. In contrast, we can optimize symbolic policies for all action dimensions simultaneously. Our approach is the first to require no prior environmental knowledge, no need for pre-trained policies, and offers a fully automated method for discovering symbolic policies. >**Q2.** Do the authors compare DSP with an implementation of https://github.com/brendenpetersen/deep-symbolic-optimization? The authors say "These methods cannot be directly applied to learning symbolic policies." It is clear that their Github page contains applications in symbolic policy learning. I'm not sure whether the statement is true or not. (1) Due to incomplete hyper-parameters of constant optimization in the DSP paper, reproducing the results is challenging. We utilized the same benchmark as DSP, allowing for a direct comparison with the results presented in the DSP paper. (2) It is important to note that the portion of symbolic policy learning in the project found at https://github.com/brendenpetersen/deep-symbolic-optimization is exactly the official implementation of DSP. Symbolic policy learning and symbolic regression tasks are distinct. Symbolic regression aims to discover symbolic expressions that best fit a dataset using a fixed function. However, symbolic policy learning involves learning from interactions within an environment and has no access to a predefined dataset or symbolic expression. Consequently, applying symbolic regression methods directly to symbolic policy learning is not feasible. For instance, DSP required extensive modifications to apply the symbolic regression method [1] to symbolic policy learning. [1] Petersen, B. K., et al. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. Proc. of the International Conference on Learning Representations, 2021.

Authorsrebuttal2023-08-11

Thanks For Your Replay!

>**Q3.** Previous works show that the CartPole and other environments have guaranteed optimal policies [1]. I think the authors should compare their interpretability with [1]. More theoretical analysis of interpretability is encouraged, as previous papers show some proof of the quality of learned policy in simple environments. A smarter way is to not claim superiority in interpretability if the experiments fail to demonstrate this advantage. I think it is better to provide comparison demos (videos or other formats) instead of human evaluations. However, Table 1 only contains results from ESPL but does not show results from related baselines. It is noteworthy that the DSP paper contains several comparisons in Table 1. (1) As this paper is in our reference, we have reviewed the previous paper [1] and did not find indication of guaranteed optimal policies for Cartpole. The best policy presented in this paper also fails to consistently stabilize the system. (2) In our global rebuttal, we provided theoretical analysis of the effectiveness of the learned symbolic policy in the Cartpole environment. (3) Furthermore, note that the claims in this paper **do not encompass improved interpretability compared to prior symbolic policy works**. This paper aims to introduce a **more efficient method** for learning symbolic policies. The interpretability of symbolic policies is inherent to the policies themselves and is not a result of the symbolic learning method. Additionally, apart from interpretability, symbolic policies are simple and computationally lightweight. (4) Since we employed the exact same benchmark as DSP and DSP provided the learned symbolic policies, due to space constraints, we did not include a direct comparison with DSP's symbolic policies in the table. Furthermore, note that DSP did not provide a comparison with previous works in Table 1 of their paper, but instead contrasted with constant optimization of DSP. One reason is that earlier work only experimented on a small set of simple environments. >**Q4.** The authors should try other discrete environments or complicated environments with rich semantics. I believe the authors should try on MineCraft [2] or other higher-level environments. These semantic-rich environments require more interpretability. The logic is, we do not need a very interpretable policy in the low-level control tasks, such as PID. However, in higher-level planning tasks, we do need more interpretability. Discovering symbolic policies in discrete action space environments and discovering symbolic policies in continuous action space environments are two different research lines. They are also not compatible in terms of methodology. Previous works (such as DSP, PRIOR, PRL) that focus on continuous action spaces often do not consider environments like 2D Minecraft. Conversely, previous works that focus on grid-based environments typically do not evaluate their methods on continuous control tasks. These prior approaches frequently incorporate domain-specific programming languages (DSLs) designed by humans in their programs to handle environmental observation inputs and action outputs. However, the research line we are part of, which employs symbolic expressions as policies, is currently not compatible with these DSLs. I acknowledge the significance of learning interpretable policies in higher-level environments. However, pursuing the research direction of learning symbolic policies in low-dimensional control tasks holds its own merit, encompassing both interpretability and efficiency during execution. Thanks again for your insightful feedback and for taking the time to communicate with us. We respectfully look forward to further discussion.

Reviewer 9VjU2023-08-11

Novelty is not very clear.

Thanks for your additional response. It seems that the novelty against DSP is the construction of action space/reward design. However, (1) I don't think this can be called a completely different RL algorithm as the core parts (the model and the algorithm) are still the same. The difference lies in the action space and reward design. However, it is hard to motivate these changes. (2) If the proposed approach trains on multiple rounds of explored experiences, is this counted in the efficiency comparisons? Also, I think the idea of exploiting existing experiences is not novel, given the vast literature on sample efficient reinforcement learning. (3) The authors admit that they do not have superior interpretability. I agree with this point and I think the draft should be revised to remove Table 1 if the results are not better in terms of interpretability. These experiments only scatter the major goal of this paper. (4) How would the baseline perform if they use the same action space/reward design as yours? Is the model of ESPL better? (5) I don't know what the authors mean by "involve constructing any additional sequential decision-making process". In fact, what can be called "additional" is vague in the context. (6) Since the baseline already uses gradient-based RL approaches, the title of this draft seems to be too broad with an overclaim of novelty.

Authorsrebuttal2023-08-11

Clarify of Novelty

Thank you again for the time and effort invested in the review process. > (1) I don't think this can be called a completely different RL algorithm as the core parts (the model and the algorithm) are still the same. The difference lies in the action space and reward design. However, it is hard to motivate these changes. **The core part (the model and the algorithm) is different.** We attempt to explain differently: DSP employs an RNN as the symbolic policy generator. Their algorithm consists of an inner RL loop and an outer RL loop (DSP Section 3, page 4). Inner RL loop: The symbolic policy produces actions that interact with the environment. Due to the absence of well-defined gradients between actions and the RNN, they construct an additional RL loop. Outer RL loop: DSP uses the RNN to gradually generate the symbolic policy (providing a symbolic operation, constant, or variable at each step). This RNN can be viewed as a policy. The reward function for the outer RL loop is the average of trajectory rewards from multiple inner RL loops. As a result, DSP cannot efficiently utilize data generated from interactions with the environment. ESPL generates symbolic policies through a differentiable framework comprising a symbolic network and a path selector. We involve only a single RL loop, which involves interactions between the symbolic policy and the environment. As gradients are well-defined between the actions generated by the symbolic policy and the symbolic network/path selector, we can directly update the symbolic policy without constructing an additional RL loop. This allows us to efficiently leverage data from interactions with the environment, leading to higher efficiency. >(2) If the proposed approach trains on multiple rounds of explored experiences, is this counted in the efficiency comparisons? Also, I think the idea of exploiting existing experiences is not novel, given the vast literature on sample efficient reinforcement learning. Our method also requires multiple rounds of interaction with the environment, which are stored in the replay buffer and can be used multiple times. Training procedures are given in the Appendix. In addition, it is because we build a differentiable symbolic policy framework that we can exploit existing experiences. > (3) The authors admit that they do not have superior interpretability. I agree with this point and I think the draft should be revised to remove Table 1 if the results are not better in terms of interpretability. These experiments only scatter the major goal of this paper. We Never claim superiority in interpretability in this paper. The purpose of Table 1 is to prove that our method is able to learn symbolic policies. >(4) How would the baseline perform if they use the same action space/reward design as yours? Is the model of ESPL better? They **can not** use the same action space/reward design. It is the proposed differentiable framework that enables the action space/reward design. > (5) I don't know what the authors mean by "involve constructing any additional sequential decision-making process". In fact, what can be called "additional" is vague in the context. In fact, symbolic policies or, more broadly, reinforcement learning is designed to solve sequential decision problems. "Cartpole" and "Hopper" are sequential decision problems. DSP also models the process of producing symbolic policies as an additional sequential decision problem. We don't need to introduce such a procedure. (6) Since the baseline already uses gradient-based RL approaches, the title of this draft seems to be too broad with an overclaim of novelty. Our approach builds a fully differentiable framework and directly uses gradients to update expressions, whereas the previous approach (DSP) introduced RNNS as a proxy to generate symbolic expressions, and they used gradients to update the proxy instead of the symbolic expression itself. We are willing to modify the title to “Efficient Symbolic Policy Learning with Differentiable Symbolic Expression” or "Efficient Symbolic Policy Learning by Directly Updating the Expression with Gradient"

Reviewer PhEA2023-08-10

Thanks to the authors for the rebuttal. I have read the rebuttal but, unfortunately, cannot find convincing results to adjust my rating. In more detail, * regarding results and evaluation, following one previous work may not be sufficient to justify the evaluation procedure. It would be better if the authors could justify more from the perspective of tasks' needs, etc. * regarding novelty, I was aware of the differences as the authors listed in the rebuttal, but do not think they are significant. As stated in my review, I consider this work as old methods/principles applied in new domains (therefore *less* novel). Overall, with these concerns in mind, I will keep my rating as 5 (borderline accept) for now.

Authorsrebuttal2023-08-13

Thanks for your reply!

Thanks for your insightful feedback and for taking the time to communicate with us. > **Q1.** regarding results and evaluation, following one previous work may not be sufficient to justify the evaluation procedure. It would be better if the authors could justify more from the perspective of tasks' needs, etc. 1. The DSP is, in fact, the SOTA in the symbolic policy domain and is the most essential baseline currently. To validate the effectiveness of our method, we employed benchmarks of DSP and compared our results against the scores presented in the DSP paper. To ensure fairness in comparison, we have to utilize the same testing methodology as employed by DSP. Furthermore, in the meta-RL experiments, for a fair comparison, we did not select symbolic policies. We adopted the same testing methodology as PEARL. 2. Due to the stochastic nature of environment initialization, averaging over multiple runs better reflects the algorithm's performance. Moreover, in the field of symbolic or programmatical policy learning, select symbolic policies from multiple independent runs is a common practice[1][2][3][4], aiding in preventing the search from being trapped in local optima. [1] Jiˇrí Kubalík, Eduard Alibekov, and Robert Babuška. Optimal control via reinforcement learning with symbolic policy approximation. IFAC-PapersOnLine, 50(1):4162–4167, 2017. 20th IFAC World Congress. [2] Verma, Abhinav, et al. "Programmatically interpretable reinforcement learning." International Conference on Machine Learning. PMLR, 2018. [3] Bastani, Osbert, Yewen Pu, and Armando Solar-Lezama. "Verifiable reinforcement learning via policy extraction." Advances in neural information processing systems 31 (2018). [4] Landajuela, Mikel, et al. "Discovering symbolic policies with deep reinforcement learning." International Conference on Machine Learning. PMLR, 2021. > **Q2.** regarding novelty, I was aware of the differences as the authors listed in the rebuttal, but do not think they are significant. As stated in my review, I consider this work as old methods/principles applied in new domains (therefore less novel). A2. Building differentiable frameworks and using gradient-based methods to obtain solutions is a conceptual approach rather than a one-size-fits-all solution. There is quite a bit of works in the NAS field alone using this approach. But it does take much effort to design algorithms for specific problems using this approach. In this paper, our novelty is reflected in the design of symbolic network structure, probabilistic path selector and corresponding training algorithm to build the first-ever differentiable symbolic policy learning framework for efficient policy learning. Thank you again for your insightful feedback and constructive comments.

Reviewer PhEA2023-08-13

I understand the authors' difficulties in experimental designs and appreciate the efforts in model designs. Nevertheless, I was aware of them while reviewing the paper. Therefore, I will still keep my rating as 5 (borderline accept). For what it's worth, in the single-task RL setting, it would be preferable and more practical to analyze the best performance among all runs or the sample efficiency to obtain good performances, from my perspective. It would thus be better to include this setting in the experiments (along with the original setting to compare with DSP). ESPL does not need to outperform DRL in numbers as it has other advantages. \ The authors had better cite the related works thoroughly as well to avoid the risk of overclaiming novelty.

Authorsrebuttal2023-08-16

Thanks for your advice!

Based on your suggestions, we conduct new experiments. For all DRL algorithms, we further tune the hyperparameters from rl-baselines-zoo and conducted three independent training runs. For each run, we select the best-performing policies across all time steps. Then we select the best performance of the three training runs as the final results. The results are shown in the following table. TD3 and SAC outperform other algorithms. For symbolic policy methods, the proposed ESPL achieves superior performance compared to DSP. ESPL is also comparable with the better-performing algorithms of DRL. We discuss the relevant work of NAS in the appendix, and we will thoroughly cite other related works in the revision. | Environment | DDPG | TRPO | A2C | PPO | ACKTR | SAC | TD3 | Regression | DSP | ESPL | |---|---|---|---|---|---|---|---|---|---|---| | Cartpole | 1000 | 1000 | 1000 | 1000 | 1000 | 1000 | 1000 | 211.82 | 1000 | 1000 | | Mountaincat | 95.36 | 93.6 | 93.97 | 93.76 | 93.79 | 94.68 | 93.87 | 95.16 | 99.11 | 94.02 | | Pendulum | -155.6 | -145.49 | -157.59 | -160.14 | -201.57 | -154.82 | -155.06 | -1206.9 | -155.4 | -151.72 | | InvDoublePend | 9347.1 | 9188.43 | 9359.81 | 9356.59 | 9359.06 | 9359.92 | 9359.25 | 637.2 | 9149.9 | 9359.9 | | InvPendSwingup | 891.48 | 892.9 | 254.71 | 890.1 | 890.11 | 891.32 | 892.25 | -19.21 | 891.9 | 890.36 | | LunarLander | 266.05 | 265.26 | 238.51 | 269.65 | 271.53 | 276.92 | 272.13 | 56.08 | 261.36 | 283.56 | | Hopper | 1678.84 | 2593.56 | 2104.98 | 2586.56 | 2583.88 | 2613.16 | 2743.9 | 47.35 | 2122.4 | 2442.48 | | BipedalWalker | 209.42 | 312.14 | 291.79 | 287.43 | 309.57 | 308.31 | 314.24 | -110.77 | 311.78 | 309.43 | | Worst Rank | 9 | 10 | 9 | 9 | 9 | **6** | 7 | 10 | 9 | **6** | | Average Rank | 5.5 | 4.125 | 6.25 | 6.125 | 5.375 | 3 | **2.875** | 9.125 | 4.625 | 3.5 | Thanks again for your time and effort invested in the review!

Reviewer PhEA2023-08-16

I appreciate the authors' efforts in rebuttal and thus decided to adjust my rating to 6 (weak accept). If accepted, I hope the authors will revise the paper as stated, including (1) a clearer and more detailed explanation of the evaluation procedure; (2) "fairer" experimental settings (even in the meta-RL setting, if doable, for the benefit of the field); and (3) a thorough discussion about the novelty.

Authorsrebuttal2023-08-17

Thank you!

We sincerely appreciate your constructive feedback and the effort you have invested during the review and discussion phases. We will carefully revise the paper according to your suggestions. Regarding meta-RL, since the previous symbolic policy methods do not support meta-RL, we compare ESPL with neural network-based approaches and adopt the same evaluation procedure of these methods which is also commonly used in the meta-RL field to ensure fairness in comparison. We will provide a clearer and more detailed explanation of the evaluation procedure for both single RL and meta RL, incorporate the experiment results and theoretical analysis during rebuttal and discussion, discuss the novelty, and make sure to cite other related works in the revision.

Reviewer uiuA2023-08-11

Thanks to authors for their insightful answers. However I still do not understand why it could not be applied to symbolic regression or classification tasks: why must be the problem be a sequential decision one to apply your method ? For me this looks similar to researches on neural network architectures (such things like [1] for instance), but with special symbolic operators at each layer. These can be applied to supervised classification tasks... I slightly improve my score (but also still think that presentation should be somehow reworked to better detail each component and give better intuition about the proposal. For instance more discussion about how are symbolic operators arranged as illustrated in fig 1 would be welcome, maybe giving ablations regarding this aspect of the contribution) [1] Tom Veniat, Ludovic Denoyer: Learning Time/Memory-Efficient Deep Architectures With Budgeted Super Networks. CVPR 2018: 3492-3500

Authorsrebuttal2023-08-13

Thanks for your reply!

Thank you very much for your positive feedback! > **Q1.** However I still do not understand why it could not be applied to symbolic regression or classification tasks: why must be the problem be a sequential decision one to apply your method A1. 1. Our framework is designed to efficiently learn symbolic policies, involving reinforcement learning processes and corresponding meta-RL designs tailored for sequential decision-making processes. 2. When talking about symbolic networks and probabilistic path selectors alone, they could be applicable to supervised learning, but it introduces additional challenges. For instance, classification tasks often entail high-dimensional inputs, whereas symbolic expressions typically require lower-dimensional inputs. This might necessitate integration with certain feature extraction methods, such as beta-VAE, and Slot Attention. Indeed, this presents an intriguing direction for future research. > **Q2.** For me this looks similar to researches on neural network architectures (such things like [1] for instance), but with special symbolic operators at each layer. These can be applied to supervised classification tasks... A2. Our approach is related to some network architecture search works. While we search for symbolic policies from the symbolic network, a series of network architecture search works construct hypernetworks and search for high-performing or computationally efficient network structures from these hypernetworks. However, our approach differs in terms of network construction, search algorithms, and the target task. We have provided a comparison with related works of NAS in Appendix E2. > **Q3.** More discussion about how are symbolic operators arranged as illustrated in fig 1 would be welcome, maybe giving ablations regarding this aspect of the contribution. A3. As described in lines 155-157, we heuristically involve more multiplication and division operators at shallow layers to provide more choice of input processed by simple operators for complex operations such as sines and cosines. We also provide an ablation study in Section 5.5. The results are presented in Table 5. Both the path selector and dense connections play a crucial role in the performance of learned symbolic policies. When the path selector is replaced with L1 or dense connections are removed, performance significantly deteriorates in certain environments. When employing dense connections and the path selector, but without symbolic arrangement (as shown in the middle structure of Figure 1), the approach can perform well in all environments. However, our heuristic symbolic arrangement further enhances the algorithm's performance. We also provide the ablation study results for meta-RL in Appendix F. We will revise the representation for better clarity. Thanks again for your time and effort invested in the review!

Reviewer iXeo2023-08-20

Response

Thank you for answering my questions, I am in favor of accepting the paper.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC