Pretraining with Random Noise for Fast and Robust Learning without Weight Transport

Summary

The authors argue that networks trained with feedback alignment can be pre-trained with random input-output pairs. They demonstrate that this allows for faster learning (after the input-output pairs). They also show that the effective dimensionality of the network activity decreases if the pretraining is used, which could explain an improved generalization.

Strengths

The paper provides a simple way to accelerate training with feedback alignment. The results are convincing in terms of faster learning, and I haven't seen a paper using random pre-activations. I think it's a good insight, and it is well presented.

Weaknesses

There is a critical missing point in my opinion. One is that the authors do not train until convergence, neither for the random pre-training nor for the task related data. I could think that there could be some limitations if the networks are trained for a long time: - For example, it could be that having a too-low rank during pre-training is bad for learning later on; since the rank is related to the expresiveness of the network this would be plausible (although it might not be the case). - It could also be that at the end of training a network trained only on data performs better at the cost of more computation (I personally don't think so, but it should be verified). Although I think I can see a performance plateau in Fig. 2 b, it should be put in logscale to see if the performance really saturated or if it can keep increasing. If it is the second, then the authors should see when does it stop increasing and whether random+data really achieves better end-of-training results. - A related point: I haven't seen the performances in numbers. While I would not care too much for them on the main text, they should be added somewhere just to clarify that the training is done to a reasonable level; if the performance is only up to 90% or something that can be achieved with a logistic regression on MNIST then there would be a problem. Note that this is not a critical flaw; it's still a good idea to pre-train for faster learning, even if that could potentially lead to slightly lower performances later on (at least where biology is concerned, 92% accuracy or 93% is not really a big deal). There are also minor issues to correct or comment: - Line 141: The loss decreased noticeably during random training. Should clarify that they mean the training loss (sorry if it is a bit pedantic) - Definition 1has a wording problem: the claim is that Wl and Bl are aligned if the expected angle decreases asymptotically during learning. Lets say that we have W^* after learning. If I just happen to have initialized the weights as W* by chance from the beginning, the statement in lines 148 and 149 would tell me that those weights are not aligned (even though they are the same). A few related works: - Error backpropagation without weight transport: While it is true that the weight transport is thought to be impossible exaclty it could be achieved by some form of hebbian-like learning, see for example Amit, Y. "Deep learning with asymmetric connections and Hebbian updates." Frontiers in computational neuroscience 13 (2019). - STDP and backpropagation: Although it is true that STDP is not enough to explain learning, there are simple architectures that can implement learning very similar to Backprop using a rate-based version of STDP (Aceituno et al. "Learning cortical hierarchies with temporal Hebbian updates"). Importantly, that not all the architectures require weight symmetry for training (the paper menitoned has a table listing them) Finally, I wanted to add a suggestion that I think would improve the paper. But it involves math and I don't know if the authors are up for that, so feel free to ignore it. I think it should be possible to show that the pretraining would reduce the effective rank. In a "batch" of pre-training with S samples and C classes, backprop would push the activity to become aligned with one of the C output directions. More analytically,: - the weight updates are Dw = x_pre delta_post, and delta_post is always one of S random vectors. - But there are only C classes, and thus the deltas within a class should be of a low rank statistically. More specifically + Compute the expectation and variance over the Jacobian, + for ReLus is the product of feed-forward weights from the current layer onwards, with a probability of 0.5 that a given neuron in the path will drop out of the derivative ). + The variance will decrease with increasing S, while the expectation will remain constant. - Once you have c delta_post, you can be reasonably sure that the result of learning will map the activity of the previous layer into something covering only the span of delta_post. + This can be done by taking approximating the averaged BP rule as a delta rule. I hope is not too unclear

Questions

I don't get what is the task agnostic section supposed to show. If the point is that the network with pre-training is better than the network without pre-training in general, then why introduce concepts such as meta-loss or task-agnostic. It seems to me that is equally valid (and maybe better) to say simply that the random pre-training is great in all tasks (and exemplify by a few).

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes. But - 314-315: I don't get what the fact that the architecture that they mention does not use weight transport implies that more experimental studies are needed. to confirm their results. What does the lack of weight transport (which is probably true) have to do with more experiments? If they want to "prove" that weight transport is not present in cortex don't worry, no one thinks it's present. Anatomically the apical projections (top-down) are different than the basal ones anyway (bottom-up)

Summary

This paper explores the idea that the brain uses spontaneous prenatal activity to optimize its structure. This is done by showing the benefits of pre-training on random noise for artificial neural networks that are trained with the feedback alignment method - a training strategy that is more biologically-plausible than traditional backpropagation. Through experiments involving networks with and without pre-training, it is shown that pre-training enables these networks to reach convergence speeds comparable to backpropagation and enables them to generalize more effectively to various tasks.

Strengths

The explanations of both the theory and empirical results are clear and well explained, for the most part. Interesting and thorough analyses were conducted to demonstrate that random pre-training of an artificial neural network trained with feedback alignment results in quicker convergence and more greater task generalization properties. I particularly appreciated the variety of studies investigating the trajectories of various metrics (e.g., accuracy, weight alignment, etc.) over time.

Weaknesses

I am unclear as to what the Definition in lines 147-149 is trying to convey. Does “cosine angle” mean cosine similarity? If so, does \mathbb{E} signify this chosen loss function, or is it an expectation over the similarities between the forward weight matrices and random feedback matrices across individual neurons? Why is \angle only used in front of the forward weight matrices? Throughout Figure 2, the authors show numerous analyses (i.e., b, c, e, and g) of how certain metrics change during training. However, the label of the X-axis says “No. of training inputs.” However, given that the various experiments outlined throughout the Appendix list the number of training epochs as 100, I am inclined to believe that that is what the authors meant instead. Further clarity in this matter would be appreciated. A minor grievance, but referring to models without pre-training as “untrained” can be misleading. While not critical to the overall message of the paper, Appendix sections C.4 and C.5 could be further strengthened by adding the training trajectories of a network trained with back-propagation for comparison.

Questions

Figure 2.b demonstrates how a network that has undergone random pre-training converges more quickly than a network without pre-training. Do these plots account for the amount of “time” spent pre-training? If not, how much would considering the extent of pre-training close the gap in training speed between these two strategies? What is the overall contribution this paper wishes to make? I am not clear as to whether this paper wishes to a) validate established neuroscience findings using artificial neural networks or b) use insights from the brain to propose a way to improve current learning strategies in deep learning. If a), I have trouble understanding how recreating findings from experimental neuroscience in an artificial neural network trained with a more biologically-plausible algorithm constitutes a novel research contribution. If b), the empirical results here indicate that models trained with feedback alignment improve as the forward weights become more similar to the fixed backward weights - a property that is established in traditional artificial neural networks via the symmetry in their forward and backward weights. Has pre-training on random inputs shown any benefits for networks trained with traditional backpropagation (e.g., faster convergence, faster task generalization, increased accuracy, etc.)? Perhaps this would be an interesting investigation that establishes a benefit of biologically-inspired learning for traditional neural networks. Answers to these questions might persuade me to move my score.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Limitations are somewhat stated and addressed. As mentioned, all of the experiments are run on a feedforward network. It could be worthwhile to verify to what extent these results are reproducible when using a different architecture, such as a recurrent or convolutional neural network.

Summary

The paper explores how pretraining neural networks with random noise can improve learning efficiency and generalization without relying on weight transport, inspired by spontaneous neural activity in developing biological brains. Key Findings: 1. Random noise training aligns forward weights with synaptic feedback, enabling more precise error backpropagation without weight transport. 2. Networks pretrained with random noise learn subsequent tasks faster and achieve performance comparable to backpropagation. 3. Random noise pretraining acts as a form of regularization, reducing the effective dimensionality of weights and leading to more robust generalization. 4. Pretrained networks show better performance on out-of-distribution tasks and adapt more quickly to various tasks.

Strengths

The paper investigates, to the best of my knowledge, a novel idea for pretraining neural networks to improve downstream feedback alignment. The paper is well written and easy to follow, experiments are nicely presented.

Weaknesses

Unfortunately I think the evidence the paper presents is not enough to back up their claims. The bulk of the experiments only studied one-hidden neural network trained on MNIST. Results and claims will most likely be overestimated and will not hold in general. The paper also does not provide any theoretical insights why this initial pre-training might help.

Questions

1) Please tune baselines. One can fool oneself very easily on the datasets and small networks you study. First, please grid search all hyperparameters in a reasonable range for the backdrop baseline and more crucially for the FA from scratch baseline. Try all common initialization schemes, and tune for them learning rate and batchsize. Please study carefully initialized random feedback weights. 2) Please study the following crucial baseline: Initialize the feedback weights B with W transpose. Your claim is essentially that when you do this you train faster than with B random. I am highly doubtful of this, and therefore your claims. I agree that your pretraining might align the weights but I am quite confident that this will not lead to better optimization. In the beginning of training weights (so only W in your case) change quickly and therefore even initial perfectly aligned weights will not bring help much. 3) How can you overcome the limitations of depth? What about layerwise local losses? If the result only hold for one-hidden layer networks, I am not confident to have this accepted at Neurips.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Please mention that your results only apply to 1-hidden layer neural networks.

Thank you!

Thank you very much for the vast additional data. This looks convinving, I will neverthless check these numbers thoroughly over the next days and get back to you. Thank you