Improving equilibrium propagation without weight symmetry through Jacobian homeostasis

Individual response to reviewer 2L2m

> *“As someone who is not in this field, I found section 2 to be informative introduction to the area.”* We thank reviewer 2L2m for the positive comment. ___ > *“While the authors suggest that hEP is potentially biologically plausible, this seems like a stretch. [...] it also seems to require that the network settle to equilibrium at multiple phases of an ongoing oscillation”* We agree that for now the question of bio-plausibility is debatable. But here are our arguments. On a conceptual level, the brain does use a hierarchy of oscillations spanning a whole range of frequency, as reviewer 2L2m pointed out, so the fact that holomorphic EP introduces oscillation in a principled way to do learning is a very promising similarity in our opinion. Yet, answering the question of plausibility for hEP will require investigating potential *mappings* from the complex units to neurons or microcircuits in the brain. For instance, ReLU activation is often interpreted as a firing rate, and a mapping to spiking network can be made based on that. For complex-valued activation, one can imagine that it encodes not only a firing rate, but also a phase with respect to an underlying rhythm. Another promising advantage that makes hEP plausible shown in the original paper (Laborieux & Zenke 2022) is the robustness to noise in the dynamics, which is not only required for any theory of learning in the brain, but also useful for neuromorphics artificial learning systems. About the question on whether the settling time is an issue for biological systems, we want to mention that neurons could use a mechanism called “prospective dynamics”, which means they try to anticipate their future activity, and it allows them to tightly track a moving equilibrium, such that the lag is reduced (see [r1], [r2] and references therein), this mechanism is compatible with our theory where neurons could track the oscillating teaching signal and reduce the time needed to learn. ___ > *“Most important for me is to explain how this could be helpful in artificial learning systems”* We list below some of the reasons why EP, and in particular the holomorphic extension could be helpful in artificial learning systems such as neuromorphic hardware [r3]: 1. The system only needs to handle one type of computation: the neural dynamics, and it need not handle a separate linear computation to derive gradients (as in BP). Designing a system that has to handle both computation with the same memory devices is more difficult than just one computation. 2. The learning rule in hEP only involves the activity of pre and post neurons, and therefore is local, which is a decisive advantage for building an efficient artificial learning system. 3. The robustness to noise already mentioned in the previous point. Not only is it bio-plausible, but also useful for noisy neuromorphic hardware. 4. Another advantage is that the error is encoded as an integral of significantly varying (non-vanishing) neuronal oscillations, rather than a finite difference between two nearby states, which is difficult to sense in noisy substrates. In short, the gradient in hEP is computed accurately despite the network oscillating in a nonlinear way, while other approaches typically rely on a linear expansion around the free fixed point. 5. In addition, the error as an integral can be accumulated in continuous time, removing the need to access the gradient at a precise oscillation period. 6. Last but not least, robustness to asymmetric weights in the extent we describe in the paper. Finally, hardware realization leveraging EP suggests that it can dramatically reduce the energy cost of AI training by several orders of magnitude [r3]. Taken together, we hope that these reasons will convince reviewer 2L2m that our work is also useful for artificial systems. ___ **References** [r1] Haider, Paul, et al. "Latent equilibrium: A unified learning theory for arbitrarily fast computation with arbitrarily slow neurons." Advances in Neural Information Processing Systems 34 (2021): 17839-17851. [r2] Senn, Walter, et al. "A neuronal least-action principle for real-time learning in cortical circuits." bioRxiv (2023): 2023-03. [r3] Yi, Su-in, et al. "Activity-difference training of deep neural networks using memristor crossbars." Nature Electronics 6.1 (2023): 45-51.

Individual response to reviewer nv6j

We thank reviewer nv6j for the positive feedback and comments. ___ > *“This paper shows the efforts in investigating the weight symmetry evolution during training in section 4. If the observations could be discussed more deeply with the figure would be better.”* We understand that reviewer nv6j means that we should discuss more in detail Fig. 4 c,g,k. For Fig 4 c,g, we record a metric quantifying the degree of symmetry of the Jacobian $J$. By expressing the Jacobian as a sum of a symmetric matrix and an antisymmetric matrix $J = S + A$, we record the quantity $\frac{\| S \|}{\|S\| + \|A \|}$ which varies from 0 to 1 where 0 means fully antisymmetric (provided $J \neq 0$), and 1 means fully symmetric. In Fig 4c we see that the metric goes very close to one because the connections are reciprocal between layers (Fig 4a). For Fig 4g, the metric increases but stays far from 1 because the connections are not reciprocal (Fig 4e), so perfect symmetry is unachievable. For Fig 4k, because we use a larger network (Fig 4i), computing the symmetry metric as in Fig. 4 c,g is not possible (the jacobian is too large a matrix), so instead we record the angle between forward and feedback weights as a replacement, for which 0 now means perfect symmetry. We see that the homeostatic loss makes the angle smaller and therefore improves symmetry, while still being significantly different from perfect symmetry. If reviewer nv6j is happy with these explanations, we will update Section 4 accordingly. ___ > *“Although ‘hEP’ is the abbreviation of ‘holomorphic Equilibrium propagation’ can be understood after reading. However, this abbreviation has not been indicated in parenthesis when mentioning its full term for the first time.”* Thanks for catching this important detail, it has now been fixed. ___ > *“The last sentence ‘It is worth nothing that…’, in Definition 1 of section 2.2, might be miswritten”* Thanks for reading carefully, the typo has been fixed. ___ > _“the sentence after Eq. (6) ‘Importantly, the quantities …, which applies only to EBMs.’, is a little longer and complex. It might confuse readers, for convenience of understanding, it would be better to split it into two simpler sentences.”_ We acknowledge that the sentence was a bit heavy, so we have separated it into two simpler sentences. ___ > *“the paragraph with Eq. (9) is intersected by Figure 2, which could be rearranged to provide enhanced readability. The same disadvantages happen in Figure 3 and Table 1.”* We thank reviewer nv6j for pointing these out, we have changed the position of the Figure 2 one page earlier so that it does not intersect the paragraph with Eq. (9). We also move Figure 3 and Table 1 so that it does not intersect the paragraph. ___ > *“The performance of the proposed method is relatively not so advanced.”* We agree that our method is tailored to EP and dynamical networks, for which the performance is behind state-of-the-art approaches that use other architectures. We reworked our introduction (in blue in the updated pdf) to justify our approach of focusing on EP primarily because it is a promising algorithm for efficient learning hardware. Despite lower performance at the moment, we believe that the results we present are necessary to further improve the performance of EP approaches in the future.

Individual response to reviewer 8uWs (1/2)

We thank reviewer 8uWs for having enough interest in our work to review it for the second time. We are also thankful for the positive comments about the writing, the high mathematical standards, as well as the value of our work. Below we reply point by point to the points raised by reviewer 8uWs: ___ > “_[...] the resulting contribution is very narrow. [...] hEP is a specialized version of EP that makes additional assumptions for complex-valued networks [...] the results only relates to a very narrow subfield [...] I wonder how useful it is to the ICLR community, beyond the very narrow sub-community that specializes in hEP_” We still respectfully disagree, and believe that our results are of interest for all the broader community studying alternative learning algorithms. For instance, we see at least three subfields different from EP, and represented at ICLR and similar conferences, for which our results are interesting: 1. Predictive coding and predictive coding networks, which are another kind of EBM often trained with algorithms related to EP (e.g. Millidge et al ICLR 2023) 2. Deep equilibrium models, which also rely on iterative equilibria for different purposes as EP, but with transferable insights (e.g. Bai, Koltun & Kolter ICLR 2021) 3. Target propagation, which is interested in mechanisms for improving symmetry (Meuleumans et al NeurIPS 2020, 2021, 2022) ___ > *“It would be very helpful to the readers and the targeted research communities if the authors motivated their choice to focus specifically on EP as opposed to alternative methods that have similar goals, but don't have the same limitations. [...] I believe that the work's value might be able to increase if the manuscript could explain its motivations and its contributions in a context broader than the EP literature.”* We focus on EP for two main reasons: 1. Because of its potential to reduce the energy cost of AI training when running on a specific hardware substrate (such as already demonstrated in Yi et al 2023), which is a broader subject than the EP literature. 2. Because EP is also a promising model for modeling learning in the brain (local learning rule, oscillations), though we do concede that a concrete mapping is still missing to make this statement compelling, which we acknowledge in the discussion. For this reason, we choose to focus on EP and reduce the impact of asymmetry and finite nudge on its performance, rather than systematically comparing it to other methods. We believe it is fair since we do demonstrate non trivial performance compared to BP for similar architectures, and on challenging tasks such as ImageNet 32. **we have updated the introduction to highlight that EP running on dedicated hardware, as demonstrated in Yi et al 2023, can reduce the training cost of AI training by four orders of magnitude**. We believe that this statement places our work in the context of reducing AI’s energy footprint, which is broader than EP itself, and also constitutes the main reason why we choose to focus on EP. ___ > *“The paper does (now) cite some of the works that had similar aims, but only in passing, only in the discussion as opposed to the motivation section, discounts the better empirical results that the other works achieved, attributing this only to the lower simulation cost of the alternatives, and does not mention that some of these methods not only perform better in classification benchmarks, but also require fewer assumptions for compatibility with biology and for efficiency in learning hardware, e.g. by circumventing the need for backward passes of information completely.”* Apology, by no means did we mean to discount other methods with our explanation for EP’s lower performance, and are sorry if it was interpreted this way. Our reasoning was that since EP has a theoretical link with gradients computed by BP, it makes little doubt that higher performances with EP can be achieved with better engineering, which is corroborated by this very recent paper [r1] that demonstrated better runtime and performance for EP. **We have now removed this argument, and updated the introduction to mention explicitly that other bio plausible alternatives well-suited for learning hardware, do perform better than EP**. [r1] Scellier et al *“Energy-based learning algorithms for analog computing: a comparative study”* NeurIPS 2023

Individual response to reviewer 8uWs (2/2)

> *“[...] it doesn't resolve the problem of asymmetry completely [...] the empirical fact that improving performance by dealing with asymmetric weights is not a new result. [...] the new theoretical results might not be very surprising [...] It should be recognized though that the new method is applicable when the connectivity is not reciprocal, whereas previous ones probably were not.”* We stress again that our claim is to mitigate the effect of asymmetry in the context of EP, rather than solving it completely. We still do believe that our way of improving symmetry for EP is novel and surprising, since it does not simply amount to making forward and backward weights identical, like in earlier solutions such as Kolen & Polack. We thank reviewer 8uWs for acknowledging that. Finally, because our new method is general, it can transfer to other subfields represented at ICLR (enumerated above). ___ > *“Could the authors please clarify, does any analog substrate achieve the relaxation "for free", or what are the requirements, more specifically? For example, the Kendall et al. reference seems to rely on a very peculiar and ad hoc hardware implementation. [...] even there, the relaxation phase would include multiple weight updates”* Sorry for the confusion. By “for free”, we meant that the laws of physics bring the substrate to equilibrium, such that the computation need not be implemented explicitly as in traditional von Neumann hardware. In Kendall et al, this is done with a nonlinear resistive network as a physical substrate, and the corresponding laws of physics responsible for doing the computation are the Kirchhoff’s laws. In particular, no weight update is required for the relaxation phase (see paragraph “Performing inference” in Kendall et al). The fact that their hardware implementation is very peculiar and hoc is on purpose. The idea is to co-design a hardware dedicated to a specific training algorithm (EP), and not a general purpose hardware. We regret the poor choice of words “for free” and **have removed it from the discussion.** ___ > *“Wouldn't alternative algorithms that do not have this relaxation phase be significantly more efficient?”* Good question. The question of efficiency is not only a question of algorithm, but also of hardware substrate on which it runs. We are not saying that EP will be more efficient than other algorithms on conventional computers, we are saying that EP is well suited to be implemented on a dedicated hardware substrate, such as Yi et al 2023, because computation can be carried out efficiently with physical laws, unlike alternative methods, to the best of our knowledge. Owing to these advantages, it can be more efficient than other algorithms running on conventional computers (cf Yi et al 2023), but we agree that this prospect is somewhat remote for now. Nonetheless, it is still a research direction worth pursuing and being represented at ICLR. ___ > *“To be clear, some of the algorithm's limitations in comparison to alternatives are: [...] multi-GPU cluster was used according to the supplementary material, despite the small networks and simple tasks [...] To their credit, the authors have mentioned some of these limitations in the discussion. They also provided some solutions or counterarguments, however these are largely theoretical, vague, or speculative.”* We are sorry that reviewer 8uWs is not convinced by our arguments. However, we do mention most of these limitations in the discussion: - Complex numbers: “oscillate in the complex plane, whose physical interpretation is not straightforward in neurobiology” - Iterative equilibria: “Another limitation of EP is that it requires convergence to an equilibrium.” - Costly simulations: “This requirement makes EP costly on digital computers.” As for the network depth, and unless we are mistaken, it is also not significantly shallower than Journé et al and Payeur et al who test on five or so layers CNNs. About task simplicity, we want to stress again that we do test on ImageNet 32, which does not qualify as a “simple task”, and is not significantly less complex than standard ImageNet, since only the resolution changes, not the number of datapoints or classes.

Individual response to reviewer EQJi

> *“introduction is a bit complex if the reader is not aware of previous work (holomorphic EP).”* Thanks for the feedback, we rewrote the sentence introducing holomorphic EP in the introduction to make it easier for the reader. It is highlighted in blue in the revised version. > *“No further questions as such. The paper has good results across different vision benchmarks.”* We thank reviewer EQJi for the positive comment.