Temporal-Difference Learning Using Distributed Error Signals

Summary

In the paper, the authors propose a novel reinforcement learning algorithm. The algorithm is based on Q-learning and uses a biologically inspired design to work with local error signals and updates, eliminating the need for the biological implausible backpropagation. The proposed algorithm, Artificial Dopamine, is motivated by the dopamine based learning system within the brain, concretely the ventral tegmental area (VTA) and the nucleus accumbens (NAc). It is realized with a novel recurrent layer/cell type, that predicates Q values from local error signals with an attention-like mechanism. The algorithm is evaluated against several baselines in a set of established RL benchmarks, and additional ablations add further insights on the algorithm. --- **Post-Rebuttal/Discussion**: During discussion with the authors and fellow reviewers, open questions were tackled, and additional results were presented. I've increased my score to 8 due to this.

Strengths

The paper tackles an interesting and important research direction with biological plausible reward-based learning, linking computer science advancements and neuroscientific findings nicely. The approach is motivated nicely, both through the background in neuroscience as well as with the connection to the recently proposed forward-forward algorithm. Similar to that, the proposed algorithm aims at training artificial neural networks without the biological implausible backpropagation. The method is introduced and explained understandable, with details necessary to understand the algorithm conceptually as well as to implement it. The included figures are of high quality and are helpful illustrations. The presented evaluations are comprehensive, solid, and appropriate for the motivating research question. Especially the provided ablations answer questions occurring while reading the paper proactively. While not covering all aspects of biological plausible learning (as mentioned by the authors), the paper shows an elegant and competitive alternative to backpropagation that is grounded in neuroscientific insights on biological plausible reward-based learning.

Weaknesses

While the majority of the paper is well written and understandable, one particular details is unclear and different sections seem to be contradicting. The AD cells/layers use multiple tanh layers, one for each action as stated and visualized. In Section 3.1, however, there is only one weight matrix W_att mentioned and used. Also in Figure 3 there is only the Q_t value, independent of an action. Inside each AD cell, are there multiple layers, one for each action, and if so, how are they used together for Q? Or is it one layer/matrix, with one dimension per action? (The first seems more plausible, but clarification and unification of notation and description would be beneficial.) While computational demanding and, hence, understandable, 10 runs are quite low for comparing RL algorithms (known problem in the community, although often neglected/ignored). Recent frameworks and metrics have been proposed tackled at these problems related to low number of runs and comparing overall algorithms performance [1]. It would be beneficial to add such metrics (e.g., performance profiles and IQM/IQR) to the paper for a better comparison of the proposed algorithm. ------------ [1] Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." Advances in neural information processing systems 34 (2021)

Questions

The algorithm is based on Q-learning, could the main ideas be transferred to other RL backbones? The reward needs to be globally available to all layers to compute the local prediction errors. Which implications does this global target induce? Any theoretical or practical drawbacks? How crucial is the choice of the two activation functions in the AD cell? Typo (?) Line 101 mentions 15 tasks, while the rest of the paper mentions 14. Additionally, the main paper actually ‘only’ uses 10 tasks, the additional 4 are only provided in the Appendix for comparison with an additional baseline. Can this be clarified or better, unified in the main paper?

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

Limitations are discussed thoroughly in a separate Section. While detailed, they focus solely on the biological background, not mentioning technical/practical limitations. Some of such limitations, like only working with discrete/discretized action spaces and the difficulty in scaling to larger action spaces, are mentioned in other parts of the paper, it would be good to add such practical limitations and assumptions in the dedicated limitation section.

Summary

This paper addresses the distributed credit assignment problem in biological reinforcement learning, where a naive implementation of backpropagation is implausible. The authors show that it is possible to update action values using a variant of the forwward-forward algorithm specialized for RL. They then validate that this algorithm is effective on a range of standard benchmarks.

Strengths

- The problem (how to implement scalable deep RL in a biologically plausible way) is important. - The paper is clearly written, with a good review of both the biology and AI background. - The results are impressive given that the system does not use backprop. - I appreciated the careful ablation analysis.

Weaknesses

- p. 5: The review by Glimcher (2011) is cited to support the claim that NAc (ventral striatum) neurons signal action value. In fact, there is a long tradition arguing that it is actually the dorsal striatal subregions (caudate and putamen) which signal action values, whereas ventral striatum signals state value (i.e., a form of actor-critic architecture; see Joel et al., 2002; O'Doherty et al., 2004). Most studies I'm aware of have shown that signals related to action value are relatively weak or absent in ventral striatum (see for example Ito & Doya, 2015). - Overall I felt that the biological evidence for the modeling framework was weak. What is the evidence that dopamine updates values locally in the NAc? This is physiologically tricky because dopamine acts postsynaptically through volume transmission, diffusing broadly in the extracellular space. What is the evidence for the particular form of hierarchical message passing architecture proposed here? Basically, I felt that the ideas here are interesting in principle but as presented are rather disconnected from empirical data. Minor: - p. 2: "connection ," -> "connection,"

Questions

- I was unsure whether the claim here is that the NAc is a multi-layer network or one layer in a multi-layer network. - If the authors would like their model to be taken seriously by neurobiologists, they need to do more to directly link it to empirical evidence. - Ideally the authors could make some neural predictions based on the model. It would also be helpful to understand how the predictions are different from those of other models. Rebuttal update: I have raised my score from 6 to 7 following the authors' responses to my comments.

Rating

7

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

The authors extensively discuss limitations in section 6. There are no potential negative societal impacts.

Summary

In this work, the Authors aimed to propose a computational model that is consistent with classical functions assigned to select regions in the reward system in the brain. To this end, they build upon George Hinton’s Forward-Forward algorithm, extending it with the ability to robustly generate continuous (instead of categorical) values and endowing it with the DQN loss. The Authors show that the proposed algorithm reaches competitive scores on a panel of standard RL benchmarks, and discuss its potential mapping on the brain circuitry.

Strengths

The text is clear and almost always straightforward. Many key statements are accompanied by reasonable disclaimers; as a result, the claims are seldom overstated. The paper has a reasonable description of the related (used) algorithms. The figures are high-quality, which makes the paper stand out among many other submissions. The experiments are performed on a sizeable panel of benchmarks. Wherever the baseline design had to be altered to ensure a fair comparison with the proposed model, correct adjustments are performed including the (hyper)parameter search for the altered models. There may be a subcommunity interested in these results. After George Hinton’s successful presentation of his Forward-Forward algorithm at NeurIPS 2022, many people present at this keynote lecture got interested in the topic.

Weaknesses

The paper’s neuro premise (Sections 1; 2.1) reads as if the roles (or even activities) of the brain regions involved in the brain circuit are known undebatable. Even though this impression is then refuted in Section 6 (which is commendable), most of the paper operates under this assumption. While the paper correctly cites some classical literature on the topic, the debates over the brain region roles proposed in these works led to more nuanced models, e.g. the mappings of the actor-critic model on the reward circuit, and then to an overall notion that such mappings represent a bird-eye view of the problem and may not necessarily be mechanistically accurate. It would be nice to explore and discuss this uncertainty, as it may alter the ground truth for this work. The paper’s comp neuro premise limits itself to the Forward-Forward algorithm. Although some of the other related work is mentioned in Section 7, these and other works have not made it into the model’s design. Meanwhile, the field of biologically plausible learning, including RL, is immense. There’s a host of work on predictive coding and local learning rules; the new works are submitted to (almost) every major ML conference. There are noteworthy works on the computational role of dopamine and RL in the brain, including the works on "Backpropamine" and DeepMind’s modeling of distributional dopamine coding. In that sense, it’s unclear why the current submission focuses on a single model, mostly validated within the ML community while ignoring the host of models validated within the computational neuroscience community. This may be mitigated by the Authors’ comment that their goal was to present a model consistent with neuroanatomy, which is valid, but brings me to the next point. A single model consistent with data doesn’t preclude the existence of multiple other models, potentially with different mechanics, leaving the chance that the proposed model has little to do with the biological reality. Indeed, a Monte Carlo – or even brute force model – will eventually learn an approximately correct Q-function and won’t need backprop / coordinated feedback signal to do so. While – and I would like to stress that – the results in this paper are correct and the wording is correct as well, arguably, to learn something from this result, one would need to perform multiple additional comparisons between the (versatile, detailed) neuroanatomical structure of the reward circuit and a panel of prominent computational models. As such comparison is not performed within the scope of this work, it is arguably too early to recommend this paper for publication in NeurIPS at this point.

Questions

N/A

Rating

4

Confidence

5

Soundness

2

Presentation

4

Contribution

2

Limitations

Related literature on neuroanatomy and computational models of the reward system are not considered. The proposed model is not contrasted against other candidate models and thus may be uninformative.

Summary

An novel, biologically plausible RL algorithm based on DQN and Fast-Forward called Artificial Dopamine is introduce. The algorithm is inspired by Dopamine pathways in the human brain and uses synchronous, locally computed per-layer TD errors for training. Additionally, a novel recurrent network architecture is proposed where layer outputs are used as inputs to the respective preceding layer at the next time step. The approach is evaluated on a range of discrete and continuous RL tasks.

Strengths

Investigating biologically plausible learning is an interesting field of research and surely significant. The manuscript is well written and reads very smoothly. Explanations of biological phenomena are clear and informative, and foremost rigorously cited. The authors clearly distinguish which parts are inspired by biology and which are deliberate choices due to empirical reasons. Limitations and societal impact are discussed thoroughly.

Weaknesses

Minor Issues: - L 116: "and is" -> "and are" - L134: "in many practical applications, such as TD learning in the brain". Makes it sound like this a proven fact that there is TD learning taking place in the brain...

Questions

- L252: "Since the local updates are distributed and per-layer, they can be computed in parallel." The temporal dependency between time-steps due to the recurrence is still present. So, does this actually translate to improved runtimes?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

2

Limitations

The Limitations such as the use of the use of a layered structure, and limited proof for the method implemented (Q-learning) occurring in the brain are discussed in detail.

thanks

Thanks for your thorough response to my comments. I hope that at least some of this will be reflected in the paper.

Discussion Period

Hi all! Just your friendly area chair checking in. First, thanks to the reviewers for your work so far on this paper. The discussion period has begun, which includes both reviewers and authors. This is our opportunity to clear up misunderstandings, get answers to questions, and generally gather enough information to make an informed decision grounded in the acceptance criteria. It seems there is general agreement that problem addressed by the paper (developing biologically plausible RL algorithms at scale) is important and that the paper is well-presented. The main critiques raised seem to concern insufficient acknowledgement of alternative neuroscientific hypotheses and biological models, and the extent of the biological evidence that this model could correspond to real activitiy/mechanisms in brains. Does that seem like a fair high-level summary? Reviewers: please carefully read the other reviews and the authors' responses (with included PDF) and let us know what follow-up questions you have and to what extent your evaluation might change as a result of the additional context. Please especially raise any points of disagreement with other reviewers or the authors, as these are opportunities for us to clarify misunderstandings and detect misconceptions.

Thanks.

Thank you for your detailed answer to my review and questions. The answers clarified the open questions and I hope to see the mentioned text changes in a potential final paper. Also waiting for "We’re working on additional figures that show the IQM and IQR of our data for our main results and ablation study, and will upload these before the end of the discussion period." for a better comparison of the methods.