Towards training digitally-tied analog blocks via hybrid gradient computation

Additional questions on the novelty of this paper.

Thank you very much for your careful and detailed response. Figure 1 and Figure 2 in the attached PDF file clearly illustrate the hardware set-up and the proposed layer design. I acknowledge that this paper is pioneering in addressing the training algorithm for hardware composed of multiple analog chips, and I agree that this is a realistic configuration. However, I still have a few questions regarding the novelty of this paper. The ff-EBM model architecture comprises a chain of the FF module and the EB module. Given that the training algorithms for the FF module and EB module are established conventions (backpropagation and equilibrium propagation, respectively), I find it challenging to identify the difficulty in designing a training algorithm for ff-EBM. Since the training algorithms for the FF and EB modules are predetermined, the primary task for ff-EBM is to ensure the proper passing of gradients between the FF and EB modules. Nonetheless, passing gradients between these modules appears straightforward by following the chain rule. Therefore, I would like the authors to highlight any algorithmic challenges related to configuring the FF and EB modules within a single network architecture. More specifically, I would appreciate it if the authors could provide more detailed explanations on the challenges involved in passing gradients between the FF and EB modules and how these challenges are addressed.

Answering additional questions on the novelty of the paper

Dear Reviewer Jt5z, We thank you very much for engaging promptly in this discussion, and are happy indeed to read that the PDF file brought some satisfying clarifications about the relevance of the problem tackled. If we understand you correctly, since BP and EP are well established algorithms in their own right, chaining them inside a given architecture appears straightforward “by following the chain rule”. In fact, we are very pleased that you have drawn attention to this issue, as we are increasingly recognizing that some of the terminology we use could prove misleading, and to some extent confusing for readers. Therefore, as requested, we highlight below the specific “algorithmic challenges” pertaining to this chaining, “how [they] are addressed” and more broadly the “novelty of this paper”. - **EP-BP chaining is intuitive, but not trivial to derive**. You write that chaining EP and BP inside a given architecture “appears straightforward by following the chain rule”. We would like to focus attention on this question: what is meant by “following the chain-rule” in the context of ff-EBMs? + Traditionally, “chain-rule” refers to gradient chaining inside *feedforward models*. Namely, let us assume a computational graph of the form $s^1 = F(x) \to s^2 = G(s^1)$. Assuming an error signal $\delta^2 = \partial_{s^2} L$, then the “chain rule” prescribes that the error signal at $x$ reads $\partial_{x}L = \partial_x F(x)^\top \cdot \partial_{s^1}G(s^1)^\top \cdot \delta^2$. + Now let us assume instead the following “hybrid” computational path: $s^1 = F(x) \to s^2: \nabla_{s^2}E(s^2, s^1) = 0$. Given that $s^2$ in this case is an *implicit* function of $s^1$, i.e. there is no explicit G mapping as before between $s^1$ and $s^2$, how would one directly apply the above “chain-rule” here? To put it differently, how do we rigorously route error signals backward through this computational graph? This is the first “challenge” we addressed. + Hence the need to derive gradient chaining inside ff-EBMs *rigorously, from first principles* by: 1) stating the learning problem as a multilevel constrained optimization problem (Eq. 8), 2) writing the associated Lagrangian (Eq. 21), 3) solving for the associated KKT conditions (Eqs. 22-34) for the primal variables (i.e. the steady states of the blocks) and associated Lagrangian multipliers (i.e. the error signals inside blocks). Therefore, this is how “we addressed” the above challenge. We emphasize that **this derivation** (namely Theorem 3.1 about the rigorous chaining of EP and BP gradients) and the **resulting explicit and implicit chaining algorithms** (Alg. 4 inside Appendix A.3, Alg. 2 in the main) **are novel**. Also, note from the above that our algorithm comes into an *explicit* and *implicit* variant (see paragraph “Proposed algorithm: implicit BP-EP chaining” L.187 and Lemma A.4), the latter appearing as a “pure” EP implementation. As such, the fact that EP-BP chaining can be cast into a *non-trivial generalization of EP*, as appearing in Alg. 2 and Lemma A.4, is not self-evident neither. - **The experimental demonstration of this algorithm is also novel**. Finally, to further highlight the “novelty of this paper”, **our algorithm was never tested in practice before this work** and having it succeed on ImageNet32 came with lots of “challenges” as well, the most important one being the simulation time. We addressed this problem in two ways: + As explained in the global rebuttal and in greater details inside Reviewer k7Rs’s rebuttal, splitting an EBM into several EBM blocks tied by feedforward modules results in an architecture that is not only more hardware realistic, but also **easier to simulate**. Indeed, instead of the superlinear scaling of the simulation time with respect to the number of layers observed in past EP works, our approach guarantees, by construction, that this **simulation time scales linearly with the number of blocks**, each of these blocks converging much faster than the full EBM counterpart. See our new table of results inside our PDF attached to this rebuttal. While the ff-EBMs trained are still relatively shallow, **they are twice as deeper as the deepest EBM trained by EP** in most recent related works. + Finally, using *Gaussian Orthogonal Ensembles* (GOE) to initialize weights inside EB blocks was instrumental in having ff-EBM training experiments work. Finally, **we would like to propose the addition of a few sentences to our introduction highlighting this novelty**, and particularly the degree to which our algorithm is derived by exploiting an intimate theoretical connection between energy based learning and **implicit differentiation**, rather than literal "backprop" as applied in standard feedforward nets where the standard "chain rule" applies. Hence our claim that this contribution belongs to the realm of EP, and *implicit learning* more broadly. Again, we thank you for drawing our attention to this.

Clarifying Scellier-Bengio's EP paper & scalability of ff-EBMs

We are very grateful to Reviewer Jt5z for spending time to understand the value of our paper, subsequently increasing our score and engaging in this discussion which will tremendously benefit the presentation of our work. Thank you so much! In the light of Reviewer’s Jt5z last answer, we would like to clarify some essential points they raised: - *"EP is designed to be applicable to any network architecture, including those with feedforward blocks"*. As mentioned in Section 2.3 of our paper, **EP only applies to energy-based models**. Fig. 1 of the seminal EP paper [1] is indeed misleading: when writing “Equilibrium Propagation applies to any architecture”, one should understand “any architecture **topology** so long as it derives from an energy function”. Indeed, as indicated by the title of the section 3 of this paper, EP really is a “Machine Learning Framework **for Energy-Based models**”: their Figure 1 is only meant to emphasize that EP applies to *any* energy-based models, not necessarily *layered* energy-based models. In this context: **layered does not meant feedforward**. As Scellier & Bengio write themselves: “*In particular, the [EP learning rule] holds for any architecture and **not just a layered architecture** (Figure 1) like the one considered by Bengio and Fischer (2015)* [which is also an EB model]”. - *"Therefore, I still view Eq. (8) as a straightforward integration of FF and EBM"*. Given the clarification above, this conclusion may no longer hold, especially when noticing that Eqs.17-18 of the seminal EP paper [1] (in the section 3 mentioned by Reviewer Jt5z), i.e. the bilevel program the EP algorithm solves, **is an explicit particular case of the Eq. 8 of our paper**, i.e. the multilevel program that our algorithm solves. - *"I think that when adopting ff-EBM, the key concern should be scalability"*. We totally agree! Investigating the scalability of ff-EBM training by our algorithm on deeper ff-EBMs, more complex tasks and exploring new datasets and architectures is part of our research roadmap. See our detailed answer to Reviewer kDhg and associated Fig. 3 inside the PDF. - *"The scalability of ff-EBM, as compared to using EBM alone, is a critical point that should be thoroughly explored"*. We are happy Reviewer Jt5z mentions the importance of this comparison since our “splitting experiment” (Section 4.3) goes exactly in this direction. This experiment reveals that a single EBM block performs comparably to an ff-EBM **of equal depth** with various block sizes. [1] Scellier, B., & Bengio, Y. (2017). Equilibrium propagation: Bridging the gap between energy-based models and backpropagation. Frontiers in computational neuroscience, 11, 24.

Source of improvement & hardware implementation of our algorithm

We are glad that Reviewer 9JF8 appreciated our rebuttal, and are indeed happy to clarify their questions. - *you propose to make the analog part to an energy model and set the digital part to the normal feedforward layer*. Yes: analog and digital parts account for energy-based and feedforward models respectively. - *you derive the automatic differentiation method to train this hybrid setup*. To be certain to be on the same page as Reviewer 9JF8, we slightly reformulate this sentence: + we apply the *Lagrangian method* to derive optimality conditions, i.e. “KKT” conditions, to “train this hybrid setup” (Appendix A.2). + the application of this method yields an algorithm which is itself a **hybrid** differentiation method which uses standard backprop (i.e. “automatic differentiation”) inside feedforward / digital parts and **equilibrium propagation inside energy-based / analog parts**. Therefore, our algorithm isn’t “pure” automatic differentiation end-to-end, but *only within feedforward parts of the model*. - *Which is the main source of the accuracy improvement?* The main source of accuracy improvement, *with respect to the past EP works*, is simply the *depth* of the model trained: the ff-EBMs trained here are twice as deep as the largest EBMs trained in previous EP works. Reviewer 9JF8’s question may then translate to: why can you train deeper networks? The most important reason, which is highlighted in Table 1 of our PDF and in our rebuttal to k7Rs, is that **ff-EBMs are faster to simulate than their fully EBM counterparts** (up to 4 times faster per Table 1). Instead of the superlinear scaling of the convergence time of DHNs (with respect to the number of layers) empirically observed in past EP works, the convergence time ff-EBMs made up of DHNs are EB blocks are guaranteed, by construction, to scale linearly with the number of blocks, the convergence time of a single block decreasing with its size. On the algorithmic side: since i) our algorithm performs on par with end-to-end automatic differentiation on the same ff-EBMs on the one hand, and that ii) ZO techniques are expected to perform less well than automatic differentiation on equivalent architectures (as highlighted in our rebuttal to Reviewer 9JF8), we can conjecture that, on *equal ff-EBMs*, our algorithm may likely perform better than ZO *when applied end-to-end*. - *“Is your proposed method hardware-friendly? Please discuss the hardware implementation of your method”*. First, as highlighted in our rebuttal, our method extends EP-based training to scaled hardware systems which: i) may not fit a single analog core, ii) may still require operations which cannot be supported on analog hardware and may instead require digital, high precision hardware. Therefore, Fig. 1 of the PDF attached to our rebuttal is a plausible depiction of the hardware implementation of our method and taking these constraints into account, our algorithm is **hardware plausible**. Second, we insist that EP-based training inside each of the analog / energy-based block inherits the “hardware-friendly” / FF-like features of EP training: in our algorithm, **gradients inside analog / energy-based blocks are computed using only forward passes / relaxations to equilibrium**. A plausible hardware implementation of this analog blocks are *deep resistive networks* [1, 2]. Lastly, feedforward blocks, which are maintained in digital, could also directly leverage quantization algorithms [3] and even **ZO algorithms** (as mentioned in L.290-291 of our paper) to facilitate their implementation on memory-constrained, low-power hardware. To summarize: our algorithm makes the best of analog and digital worlds, by: i) being hardware plausible at the system level (Fig .1 of PDF), ii) preserving the “hardware-friendliness” of FF-like learning inside EB/analog blocks, iii) possessing the ability to leverage any quantization algorithm inside feedforward blocks and iv) possessing the ability **to apply ZO algorithms** instead of backprop (as currently done with our algorithm) **inside feedforward blocks**. Finally, we would like to thank reviewer 9JF8 for drawing our attention to some critical points where our paper could better communicate some of the high-level motivations and details of our algorithm. In addition to what was proposed in our original rebuttal, we would like to propose adding a pseudo algorithm in the appendix showing how ZO could be applied within feedforward blocks (instead of backprop currently) such that gradients would be computed **everywhere with forward passes only**, i.e. both inside analog blocks (by EP) and feedforward blocks (by ZO) (as first suggested in L. 290-291 of our paper). [1] Kendall et al (2020). Training end-to-end analog neural networks with equilibrium propagation. [2] Scellier, B. (2024). A Fast Algorithm to Simulate Nonlinear Resistive Networks. ICML 2024. [3] Lin et al (2022). On-device training under 256kb memory. NeurIPS 2022