Training biologically plausible recurrent neural networks on cognitive tasks with long-term dependencies

Summary

The paper proposes a method for extracting long-term dependencies in biologically plausible leaky RNNs. The motivation of this work is well-presented and has contribution to the neuroscience field.

Strengths

Strengths: * The paper demonstrates significant improvements in terms of performance when compared to other baseline methods. * The paper is well structured and has a smooth and concrete flow. * I find the incorporation of biological domain knowledge in the algorithm quite interesting and insightful.

Weaknesses

Weaknesses * In terms of presentation, it would be helpful to list the properties of datasets used in the experimental part. * Comparison with LSTMs and GRUs-based methods should be a part of the experiments in order to prove the superiority of the suggested approach.

Questions

Please consider the things listed in the “Weaknesses” section. Also please consider providing information regarding any potential future improvements.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

To my knowledge and understanding, there is no potential negative societal impact of this work. Other limitations please see “Weaknesses”.

Summary

The authors suggest a method for accelerating the training of continuous time vanilla recurrent neural networks. Specifically, they add temporal skip connections during training, and gradually remove them throughout the training process. For tasks with long term memory requirements, this can provide up to 3 times less training steps. The authors also numerically examine different variants of such skip connections. In particular, they use an interpolation between coarse-grained and fine-grained dynamics.

Strengths

The authors highlight an existing problem with RNNs and offer a novel way of addressing it.

Weaknesses

The main claim – faster training – is not verified. Networks with skip connections might be more computationally expensive. Table S3B gives a hint that there is some saving, but this saving is quite modest. It’s not clear what is the relationship between the algorithms and compute time, and how this scales with parameters such as network size. It was hard to understand the exact form of the equations, due to the use of “before” and “after” labels. I assume (also from figure 3) that the meaning is that the high-resolution dynamics runs in parallel, and the two versions are combined. If this is correct, then equations 7 and 8 seem incorrect. Equation 7 is the case for CD, not for the discretized equation 1. The last term of equation 8 (“before”) also depends on the recurrent connectivity. A rough approximation would probably be $W^\theta$, because W is applied $\theta$ times to generate this term. If the eigenvalues of W are larger than 1, this will amplify them. The relation to prior literature is very partial. For instance, skip connections are only described in the supplement – without references in the main text. Other approaches for training RNNs on long dependencies (a few references below as an example) are also not discussed. Trinh, T., Dai, A., Luong, T., & Le, Q. (2018, July). Learning longer-term dependencies in rnns with auxiliary losses. In International Conference on Machine Learning (pp. 4965-4974). PMLR.Le, Q. V., Jaitly, N., & Hinton, G. E. (2015). A simple way to initialize recurrent networks of rectified linear units. arXiv preprint arXiv:1504.00941. Arjovsky, M., Shah, A., & Bengio, Y. (2016, June). Unitary evolution recurrent neural networks. In International conference on machine learning (pp. 1120-1128). PMLR.

Questions

The use of the term biological throughout the paper is somewhat misleading. It is true that notations vary between fields and subfields, but still – biological is usually reserved for actual biological networks. Terms such as vanilla or continuous time are perhaps more appropriate. The term biologically plausible could also be used, but that is usually used for adding properties above simple rate networks – separation to E and I neurons, spiking, dendrites, short term plasticity, excitability and so on. Line 106: alpha used without definition Figure 1: perhaps use log scale for X axis of panels B,D. Colors in figures are hard to discern (the specific shades of blue and green used). If I understand correctly, skip connections usually come with trained weights. In this work, they have the untrained weight beta, to allow their eventual removal. This should probably be stressed. Comparison of true and modified dynamics (line 170, Fig 3B) – I couldn’t fully understand the setup. Were these networks with the same random seed? Why should neural activity be identical? There are many symmetries, such as permutations, that are expected to arise while training. Was some alignment done between the neural trajectories? Figure 4. The Y axes are very different in every panel. If possible, this should be avoided. If not (different scales of results), this should be stressed to the reader. Figure 4B – how do these results look when splitting the tasks? Figure 4B, last panel. Is the bimodal distribution a result of different tasks? Figure 4B the X labels are not intuitive. The first paragraph of the discussion is titled “existing methods” but has no references.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

yes

Summary

In this study, the authors propose a couple of architectural innovations to train biological RNNs in a stable manner on a suite of 16 cognitive tasks, some of which require the integration of long-range temporal dependencies. The authors motivate their work from the lack of cortical evidence for gating mechanisms that are crucial to high-performing RNNs such as GRU and LSTM. Instead the authors focus on incorporating skip connections through time in various implementations to learn long-term dependencies. Over various experiments and numerical simulations of recurrent trajectories, the authors show that their proposed DASC method to apply skip connections through time performs effectively, especially when the task requires learning long-term dependencies.

Strengths

+ RNNs are widely used in encoder-decoder models trained on cognitive tasks or neural data. The proposed work aims to address one of the issues that plagues RNNs - learning long range dependencies. Skip-connections through time aren't actively explored in RNNs for computational neuroscience and I appreciate the authors' effort in studying this mechanism for improving RNN training stability. + The figures in the paper are commendably presented and really help the readers to understand the underlying mechanisms and differences between Coarsened Discretization, Skip Connections Through Time and DASC.

Weaknesses

- My main issue with the paper is that I believe it is not well motivated; they mention that using biological RNNs without gates is important but this claim is not backed with any advantage of biological RNNs over ones with gates. As such, I don't think one would consider gating as biologically implausible as this computation has surfaced in various parts of the cortex. - A related issue is that, the authors have not shown comparisons of their proposed skip-connections models against the established gated recurrent networks like GRU and LSTM. I find this as an important comparison that is missing, as it allows for relatively measuring the contribution of temporal skip-connections against gating mechanisms. - I believe the writing in the paper could be improved further, there is a lot of information that isn't central to the paper's theme such as Section 2.2 (Review of model parameters) which obstructs the paper's flow and makes it harder to glean the main message. - Even though the authors mention a variety of usecases for biological RNNs (such as modeling neural responses), they only evaluate their models on a suite of simple cognitive tasks. It would be more holistic if the authors also included observations on whether the alternate usecases of RNNs in neuroscience also benefit from adding dynamic skip connections through time.

Questions

- My suggestion to the authors is to strengthen their stance against gated recurrent networks; if these gated networks are good models of cognitive tasks and neural data, why should they not be favored over biological RNNs? As such, all these networks are trained with BPTT which also does not have strong biological evidence.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors have stated limitations of the work adequately.

Summary

This paper describes and tests several methods for training a class of “biologically-plausible” recurrent neural networks (RNN), which are discrete approximations of continuous-time leaky RNNs. This type of gate-less RNN model has difficulty learning dependences over long time intervals, so tweaks are applied during training time to help stabilize and propagate gradients through time. Three such tweaks are explored: coarsened discretization (CD), where the RNN time step is simply increased during training; skip connections through time (SCTT); and the new method dynamics-aligned skip connections (DASC), which cleverly combines CD and SCTT. Some theoretical analyses of these methods are provided, and then each method is evaluated on a variety of realistic cognitive tasks. The results show that the new method, DASC, tends to perform best (measured by how long the model takes to reach a performance criterion), although different tasks require different hyperparameters.

Strengths

* This paper is quite comprehensive, including clear mathematical descriptions (and nice figures) of the methods, theoretical analyses of their properties, and thorough experimental evaluation. * The overall approach—modify the network at training time, then remove the modifications after training—is a clever end-run around issues of biological plausibility. The resulting models are always in the “biologically plausible” class, even though the training procedure that arrived there is not. (And we suspect that training with backpropagation is implausible anyway, so no big change there.) While I’m not sure I’ve seen this approach taken with this type of RNN in the past, the authors might note that similar ideas have been floated for spiking neural networks by using surrogate loss at train time. * The results are solid and seem useful going forward!

Weaknesses

The science in this paper is strong, so all the weaknesses noted here relate to clarity and presentation. * I found many points in this paper fairly difficult to understand or under-explained until I read the Appendix. There is quite a bit of information in the Appendix that, by any right, should be in the main paper. (References to earlier skip-connection work, the actual model formulation used, how training and evaluation was performed—it was unclear from the main paper how or when the extra discretization and skip connections were removed when testing models!—and descriptions of the tasks.) I understand that the authors are operating under strict page limits, but should this paper be accepted I strongly urge the authors to consider moving some things that are likely critical information for a reader into the main text. * While the figures illustrating the model (Figures 2 & 5) and showing experimental results (Figure 4) were fairly clear, those summarizing earlier work (Figure 1) and illustrating the modified model dynamics (Figure 3) were much more opaque. I’d recommend the authors attempt to simplify or clarify these figures. * One issue I was left somewhat uncertain of was novelty. As mentioned above, skip connections are by no means new, but are presented as a novel technique in the main text of this paper. I am not deeply embedded enough in the literature to know how to place DASC. I would have felt better about this if the authors had spent a bit more time & ink on describing earlier approaches to these issues and connections to their proposed methods.

Questions

None noted. (See suggestions under "weaknesses" above.)

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

2 fair

Contribution

3 good

Limitations

The discussion of limitations is clear and complete.

Summary

This paper proposes a way to speed up biological RNN training by up to 3x using well placed skip connections during training time.

Strengths

This paper will be of interest to researchers training biological neural networks, especially for hard cases where a 3x time speed up can be achieved. Presentation is clear and the work should be relatively easy to reproduce.

Weaknesses

I find a 3x speed up only for really difficult cases to be of modest improvement. Given currently powerful computers, training a biological style small recurrent neural network is relatively fast which reduces the practical significance of this work.

Questions

Does the training procedure affect the type of synaptic connectivity solution learned? This should be a interesting direction to explore.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

none

More questions

Thanks for the answers and clarificaions. A couple of questions Regarding the equations: Figure 2D shows multiple steps that are later combined with a single step. The equation you wrote in the rebuttal is a single step. Can you please clarify? I might be missing something here, but there seems to be an inconsistency. Literature: I think curriculum learning is similar in spirit to this work, because there is a modification to the training stage.