Correlative Information Maximization: A Biologically Plausible Approach to Supervised Deep Neural Networks without Weight Symmetry
The backpropagation algorithm has experienced remarkable success in training large-scale artificial neural networks; however, its biological plausibility has been strongly criticized, and it remains an open question whether the brain employs supervised learning mechanisms akin to it. Here, we propose correlative information maximization between layer activations as an alternative normative approach to describe the signal propagation in biological neural networks in both forward and backward directions. This new framework addresses many concerns about the biological-plausibility of conventional artificial neural networks and the backpropagation algorithm. The coordinate descent-based optimization of the corresponding objective, combined with the mean square error loss function for fitting labeled supervision data, gives rise to a neural network structure that emulates a more biologically realistic network of multi-compartment pyramidal neurons with dendritic processing and lateral inhibitory neurons. Furthermore, our approach provides a natural resolution to the weight symmetry problem between forward and backward signal propagation paths, a significant critique against the plausibility of the conventional backpropagation algorithm. This is achieved by leveraging two alternative, yet equivalent forms of the correlative mutual information objective. These alternatives intrinsically lead to forward and backward prediction networks without weight symmetry issues, providing a compelling solution to this long-standing challenge.
Paper
References (70)
Scroll for more · 38 remaining
Similar papers
Peer review
Summary
This paper seeks to present a biologically plausible learning approach for supervised learning in deep neural networks. Unlike backpropagation, the approach does not require symmetric weights in the forward and backward directions. The approach relies on an information theoretic approach which seeks to maximize mutual information between layers in the forward and backward direction. The approach is demonstrated on simple data sets (e.g. MNIST) as well as on 3-compartment models of pyramidal neurons.
Strengths
The manuscript addresses an important problem--essentially how can supervised learning be implemented in biological neural networks. It proposes a solution to the well-known weight symmetry/transport problem. And it seeks to do so in a principled way using information theoretic notions
Weaknesses
Unfortunately, the paper is poorly written, with heavy notations and equations which often obscure the approach rather than clarifying it. There is no clear expression for the learning algorithm. It is hard to see that the learning algorithm is local both in space and time, which is a major requirement in a biologically plausible network. For the experiments, the authors report the test accuracy. However other metrics would also be interesting, for instance, the degree of symmetry between the final weights in both directions. It seems that the approach still requires propagating error information over long distances (across many layers) which may also be problematic from a biological point of view. Supervised learning is not particularly biologically plausible. This point should be addressed, as a minimum using self-supervised learning in combination with the proposed approach. The authors should mention more clearly that the weight transport problem is completely solved by random backpropagation or feedback alignment. Thus the advantages of their approach, if any, should be contrasted with feedback alignment.
Questions
It is not clear what you do at the top layer with the targets and the error. Are you clamping the output layer to the targets and then applying your backward pass? The term "disputed" in the abstract is too weak. I think there is a strong general consensus that plain backpropagation is not biologically plausible. line 42: "maximize the linear dependence"--why linear, since the networks are typically non-linear? furthermore it is easy to maximize linear dependence by making the signals identical (i.e. identical activities in two neighboring layers of the same size, which is NOT interesting. line 59: "predictors" of what? Figure 1 is not easy to understand at this stage of the paper. In fact, the use of compartmental models at the beginning of the paper is confusing. Maybe these should be moved entirely toward the end of the paper or in an appendix to improve readability. To the best of our knowledge, the result by Liao et al is not reliable --it has not been confirmed systematically. The term polytopic is not very common and should be defined. Is this assumption needed in the case of artificial neural networks? line 148: why do you mention that the objective is stochastic, and stochastic in which sense? Also, to be clear, you should specify whether you are trying to maximize or minimize the objective. Is there some Gaussian assumption behind equation 2 (if so it should be stated explicitly). Also, any connection or divergence from better-known concepts, such as mutual information, should be clarified. Equation 2 and what follows are particularly unclear and hard to follow. 153: "problem" Which problem exactly? page 6: the learning rule is not clear. Can you write it in the standard form of "\delta w_{ij}=learning rate x update urle....$?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
See some of the remarks above. There is no discussion of the limitations of the proposed approach.
I agree that the reviewers have answered a fair number of comments--hard to tell something more precise without seeing the revised version. I am happy to move my score up by 1.
Summary
The authors present a novel strategy for learning in neural networks. In particular, the authors derive update rules for neurons/synapses which maximises the correlative information between layer activations. This strategy avoids the weight transfer problem, and naturally gives rise to a biologically emulating architecture of multi-compartment pyramidal neurons with lateral inhibition.
Strengths
- the authors present what seems a mathematically sound and creative strategy for credit assignment. Without extensive knowledge in this area, the derivation of update rules seems original and of good quality - The resulting likeness to a multi-compartment model with lateral inhibition is interesting - the text is generally well written (though the presentation itself is dense, see below)
Weaknesses
- in general I found the paper very dense - I personally think 27 equations is too many for a main text. I appreciate that the main contribution of this paper is analytical, but the think the authors would do well to sacrifice some of the less key equations (move to SM) to make space for additional intepretation/experiments - As stated above, I would have liked to have seen more intepretation and experiments with respect to the model. For example, what predictions does the model make in terms of the balance of bottom-up/top-down signals? How does this change over learning? How does it compare to biology? Same for interneurons - The actual performance of the model does not seem too impressive, at least compared to standard backprop (e.g. on the CIFAR-10 dataset). Moreover, given that a key property of the model is to avoid the weight symmetry issue, I would think it sensible to compare the model to backprop with random feedback weights (feedback alignment). - I think the authors coud make it more explicit what are the differences between their model and the model in Golkar et al. 2022. In particular, explicitly highlighting the similar and new terms when presenting the mathematical formulation
Questions
- The segway at line 34 was unclear to me - I found section 2.1 confusing to read because the equation which derives the activity r^k of a given layer is not expressed. Is this intentional to keep it general? It's confusing to understand whether the activities between the layers have a relationship at all at this point. - line 140: the inequality used to express the hypercube is not well defined for vectors - The CMI metric (equation 2) is a complicated equation with determinants and auto/cross-correlation matrices. I would have liked an intuitve (perhaps geometric) description of this measure - line 150: R_{r^k r^{k+1}} hasn't appeared yet but is already being described - in section 2.3.1 the variable s is introduced without any explanation as to what it represents. Is it time? i.e. u[t, s] is the t dataset example at time s? - line 231: I didn't understand the decomposition of M. Where does D come from and why does it mirror autapses? Is it the identity matrix in the expression of M? WHere does the negative O part (interneuron) come from? - equations 26,27: sorry if this is a naive question, but why can't the weights just be updated directly using equations 23,25?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
I would recommend a limitations section (or at least more discussion). For example, I would be interested to know if the sensitivity of the model to hyperparameter choices is high, or whether there is a strict need for the feedback matrix at the last layer to be the identity.
Summary
The authors introduce a biologically plausible training paradigm for a deep neural network that sidesteps the weight transport problem while achieving competitive results. Their approach is normative, in that both the network's architecture as well as its learning rules can be derived from an information maximization approach. The asymmetry between forward and backward weights is achieved by leveraging two different formulations of the inter-layer correlative information.
Strengths
*Originality:* The work provides a novel approach for deriving biologically plausible strategies for learning in deep neural networks. *Quality:* The paper contains a significant amount of work to support its findings. Importantly, both theory and computation are used in tandem. *Clarity:* The presentation is mostly clear, but some significant explanations are missing or too sparse. See below. I also want to praise the authors for including the code that they used with the submission (something that I believe should be true for all papers, but sadly is not). *Significance:* The work is significant for neuroscience because the learning algorithms used by the brain are not yet understood. Having a good grasp over the range of possible mechanisms that biology could have used to train natural neural networks is essential to allow experimentalists to probe what choice(s) is (are) actually used. The work is also of potential significance for machine learning, since the algorithms used by the brain might provide advantages over the gradient descent with backpropagation methods used to train artificial neural networks.
Weaknesses
1. The correlative mutual information metric requires a bit more discussion. The regularization coefficients $\epsilon_k$ appear in eq. (2) but are not discussed at all until much later, and even in the derivation in Appendix A, the need for this regularization is not explained. On first guess, the need for $\epsilon_k \ne 0$ is due to having a low rank covariance matrix $\mathbf R_{\mathbf r^{(k+1)}}$. However, this seems inconsistent with the importance of these coefficients in the network dynamics and learning rules. This requires a more detailed discussion in the main text, and especially in the Appendix. (If space is an issue, I suggest removing most of lines 170-174, which are almost identical to 149-153; it can simply be stated that the sample covariance matrices from eq. (6) need to be used instead of their exact counterparts to get online training rules.) 2. Related to the regularization coefficients, I am a bit perplexed by eqns. (10), (11). The Taylor expansion in these equations is performed around the identity, but that makes the expansion parameter be $1 / \epsilon$. Since $\epsilon$ is small, $1 / \epsilon$ should be big, making it hard to justify ignoring subsequent terms in the Taylor expansion. This needs to be explained in detail. 3. The jump to the dynamics equations (15)–(17) is too abrupt. Either an explanation should be provided or a reference to a relevant Appendix section.
Questions
Questions * for me, phrasing the method in terms of using two *equivalent* forms of the correlative mutual information is misleading: if they are equivalent, how could they lead to the desired asymmetry between feed-forward and feed-back weights? I may be wrong, but I believe that the resolution lies in the Taylor expansion around line 178 -- it's not the differing exact expressions that lead to the asymmetry, but the different *approximations* * the definition of $\mathbf M$ on line 212 is a bit confusing. Should the factor of 2 apply to both terms? I would have thought that the leak term in the expression for $\mathbf M$ is supposed to counteract the leak term in eq. (15) as long as $\mathbf u$ is inside the feasible domain * also regarding $\mathbf M$: since $\epsilon$ is small, my understanding is that $\mathbf M$ is also small; in this case, I would suggest leaving $\epsilon$ out of the definition for $\mathbf M$, to make the scale of each term more apparent; e.g., in eq. (18), I would imagine $\mathbf W_{fb}$ has the leading contribution to $\mathbf v_A$, while the $\mathbf M$ term is a sub-leading correction * were hyperparameters optimized for each task for all of the algorithms in Table 1? Minor comments: * in eq. (12), the notation $\hat J_k$ is used but it had never been introduced before. Please define $\hat J_k$ first before using * the update rule for the lateral coefficients (lines 280-281) should be written in terms of the biologically motivated parameters ($\mathbf M$, or even better, $\mathbf D$ and $\mathbf O$) instead of $\mathbf B$. * in the numerical experiments section, please include at least basic details about the networks that are used (e.g., number of layers) * below eq. (A.12), the clipping operation $\sigma_+$ is invoked out of nowhere. I think the point is to show how this operation can be justified as a way of enforcing the KKT conditions; this should be made clearer * the plots in Appendix Figures 7–8, 12–14 are very sparse and it's not clear what information we are to glean from them; I suggest combining some of these, e.g., compare the accuracy attained by different network architectures and / or activation functions * in the studies from Appendix F, the range over which the hyperparameters are varied seems too small because the variation in accuracy is very low, barely beyond the variability over different runs; I suggest using larger variations to show the trend
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
The authors have adequately discussed limitations of their work.
Summary
The paper proposes Correlative Information Maximization as an underlying objective for biologically plausible learning. The objective produces a multi-compartmental neuron model, and can operate with feedback connection that are plastic, but not tied to the feedforward ones.
Strengths
The (approximation to the) CorInfoMax objective produces a tractable model of a neuron with several compartments. This resonates with previous ideas of credit assignment with apical dendrites, and (I guess) generates experimentally testable predictions due to the specific interactions between compartments and weights. The approach to weight symmetry is interesting and might implicitly lead to weight symmetry (although see weaknesses). Overall, this is a novel idea, even though it is very related to previous works that use apical dendrites/predicting coding/etc. as a mechanism for credit assignment.
Weaknesses
The experiments in Tab. 1 have multiple problems. There's no comparison to backprop and no explanation of used architectures in the main text. Presumably the architectures were pretty small, given poor CIFAR10 performance. Related, all experiments show feedback alignment-level performance (i.e. good on MNIST, OK on CIFAR10 for a small network that reaches about 50% accuracy). Thus, we can’t draw any conclusions about the effectiveness of this approach without considering at least larger networks and maybe harder datasets (as feedback alignment doesn't scale beyond those cases). The minimum aim would be to train a standard ResNet18 on CIFAR10 with backprop (should be around 90% accuracy), and compare it to all algorithms in Tab. 1. The authors also missed an important previous work -- Deep Learning without Weight Transport by Akrout et al. (2019). That paper proposes a simple mechanism for the weight transport problem that is a bit different from the one here, but it is still worth discussing in the context of backprop approximations/alternatives.
Questions
Main question: are the feedback weights different from the feedforward at the end of training? I think it is an interesting question in itself, since "no" would mean your algorithm approximates backprop (similar to predictive coding approaches?). If they are different, I wonder if that could hurt performance on harder tasks and if that could be fixed somehow (akin to weight symmetry in Akrout et al.?) Is CorInfoMax here similar to the Information Bottleneck ideas? The abstract says > The backpropagation algorithm... it remains an open question whether the brain employs supervised learning mechanisms akin to it Backprop doesn’t imply supervised learning (see VAEs, self-supervised methods and so on). The overall claim is fine, but it shouldn't be about supervised learning. Eq. 27: should the second $e^{k+1}$ be $e^k$? **Overall**, it is an interesting contribution but the experiments are very small-scale and the proposed approach is not compared to the main competitor, backpropagation. ---------- **Post-rebuttal**: feedback alignment-level performance is a limitation of this work, but the principled approach to derive multi-compartment models and additional evaluations done during the rebuttal justify an increase of the score from 5 to 6.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
The limitations and potential impacts have been addressed.
Response to rebuttal
Thank you for the response! Overall, I appreciate the clarifications and additional evaluations. However, the additional experiments confirm my concern in the original review -- the method scales similarly to vanilla feedback alignment (FA). This is evident by similar performance, and a large performance gap between CorInfoMax/FA and backprop in Tab. 2 of the rebuttal pdf. While a performance gap is not a bad thing per se, I suspect the method will fail at hard tasks just like FA. And I also suspect it could be fixed by explicitly aligning the feedback weights with the feedforwards ones. Since fixing the weight transport problem was one of the main goals of the paper, I think presenting a more capable method than FA is crucial. > We deliberately excluded CNNs, including ResNets, due to their weight-sharing feature that doesn't align with locally connected neuronal models in the brain. Many papers on biologically plausible deep learning use CNNs though, so including those results might help to show that the algorithm is capable of learning hard tasks (without implying that this is how the visual stream works). I'm open for a discussion, but currently I remain skeptical of the method's value for the field.
Response to Reviewer MCJo's Response
Thank you once again for your thoughtful feedback and the time you’ve invested in evaluating our work. We genuinely value your concerns regarding the performance on more challenging tasks. We acknowledge that achieving high performance on hard machine learning tasks remains a collective challenge for all biologically plausible models. However, this may not be the most relevant and only evaluation metric from the perspective of explaining how brains work. In fact, matching biological reality and interpretability via the use of normative principles are important criteria (See, e.g., [a,b,c,d,e,f] below) . Indeed, the primary contribution of our article lies in its principled approach: we introduce a normative framework grounded in information theory. Biologically plausible networks comprised of multi-compartment neurons with both recurrent and asymmetric feedforward/feedback connections naturally emerge as solutions of the optimization settings put forward through this framework. At the same time, our approach provides principled interpretations of lateral, feedback, and feedforward connections, as well as activation functions. We can use information theoretic lens to interpret the role of these network components in terms maintaining bidirectional information flow, avoiding embedding space degeneracy (through lateral connections and interneurons) while achieving compression by eliminating redundancy (through feedback/feedback connections) and domain constraints (activation functions/interneurons). At the same time, resulting networks achieve similar performance to existing biologically plausible benchmarks without weight reuse. We are confident that the foundational nature of this framework, when extended and combined with additional biological and normative constraints, has the potential to address more practical concerns in the field. [a] Golkar S, Lipshutz D, Bahroun Y, Sengupta A, Chklovskii D. A simple normative network approximates local non-Hebbian learning in the cortex. Advances in neural information processing systems. 2020;33:7283-95. [b] Meulemans A, Zucchet N, Kobayashi S, Von Oswald J, Sacramento J. The least-control principle for local learning at equilibrium. Advances in Neural Information Processing Systems. 2022 Dec 6;35:33603-17. [c] Alonso N, Millidge B, Krichmar J, Neftci EO. A theoretical framework for inference learning. Advances in Neural Information Processing Systems. 2022 Dec 6;35:37335-48. [d] Song, Yuhang, et al. “Can the brain do backpropagation?---exact implementation of backpropagation in predictive coding networks.” Advances in neural information processing systems 33 (2020): 22566-22579. [e] Bredenberg C., Savin C. Desiderata for normative models of synaptic plasticity, arXiv:2308.04988, 2023. [f] Lipshutz D, Bahroun Y, Golkar S, Sengupta AM, Chklovskii DB. A normative framework for deriving neural networks with multi-compartmental neurons and non-Hebbian plasticity. PRX Life 2023.
Response to authors; score increased to 6
Thank you for the response! > We are confident that the foundational nature of this framework, when extended and combined with additional biological and normative constraints, has the potential to address more practical concerns in the field. I ultimately agree with your points, especially the ones concerning a derivation for multi-compartment models. I still think FA-level performance is a limitation of this work, but given the discussion here and with other reviewers, I think the work is interesting enough and increase the score from 5 to 6.
Response to Reviewer Comment
We would like to thank you once again for the valuable feedback and interesting questions. Your raised concerns have been instrumental in enhancing the clarity of our paper, and we deeply appreciate your positive assessment.
Thanks for the explanations. It seems like the answer regarding the $\epsilon$ terms is that they are actually relatively *large*, not small. I suppose this is a valid choice, although I'd be curious to know whether the quadratic term in the Taylor expansion is actually negligible compared to the linear one that you keep. Regardless of this, such a large value for $\epsilon$ goes well beyond regularization in the sense of avoiding divergences, it appears to act more like a prior, pushing your system in a direction that presumably is desirable for some reason. What is that direction and why do you want to bias the system like this? In other words, by using a large $\epsilon$, you are not quite optimizing correlative mutual information anymore, and in that case, you should explain what you are optimizing instead, and why.
Response to Reviewer Comment - Part 1
Thank you for your comments and questions, which enhance the understanding of our article. Before directly addressing your query, we will first delve deeper into the discussion on the $\epsilon$ parameter, and then offer our responses: Upon examining the expression for correlative mutual information (CMI), $$\overset{\rightarrow}{{I}^{(\epsilon_k)}}(\mathbf{r}^{(k)}, \mathbf{r}^{(k+1)}) = \frac{1}{2} \log \det \left(\mathbf{R}\_{\mathbf{r}^{(k+1)}} + \epsilon_k \mathbf{I}\right)- \frac{1}{2} \log \det \left(\mathbf{R}\_{\overset{\rightarrow}{\mathbf{e}^{(k+1)}\_\*}} + \epsilon_k \mathbf{I}\right), \quad (2)$$ $\epsilon_k$ appears to function as a correction factor to compensate for rank-deficient correlation matrices of degenerate random vectors. From this perspective, this adjustment serves two primary purposes: * To establish a finite lower bound for the entropy, and * To circumvent numerical optimization issues, given that the derivative of the $\log\det$ function is the inverse of its argument. In fact, robust matrix factorization methods that rely on determinant-maximization use this perturbation for aforementioned reasons [a]. Additionally, a recent study links $\epsilon_k$ with a parameter of the **Inverse Wishart prior** distribution on the covariance matrix of the row vectors of one of the factors [b]. Moving beyond these interpretations, we first observe that (2) defines **a family** of correlative mutual information definitions. **For each $\epsilon_k$ choice, we have a valid alternative correlative mutual information definition [35]. Hence, we do not necessarily interpret them as approximations of $\epsilon_k=0$ case, instead they are alternative CMI measures.** To see the impact of $\epsilon_k$ choice: -We note that the prediction error covariance matrix in (2) $$\mathbf{R}\_{\mathbf{e}^{(k+1)}\_\*}=\mathbf{R}\_\mathbf{r^{(k+1)}} - \mathbf{R}\_{\mathbf{r}^{(k)} \mathbf{r}^{(k+1)}}^T(\mathbf{R}\_\mathbf{r^{(k)}} + \epsilon_k \mathbf{I})^{-1} \mathbf{R}\_{\mathbf{r^{(k)}}\mathbf{r}^{(k+1)}} \hspace{0.2in}(A)$$ corresponds to the error correlation matrix for the best linear regularized minimum mean square estimator of $\mathbf{r}^{(k)}$ from $\mathbf{r}^{(k+1)}$. This estimator is obtained as the solution of the optimization problem \begin{eqnarray} \underset{\mathbf{W}\_{\mathbf{r}^{(k+1)}|\mathbf{r}^{(k)}}}{\text{minimize }} {E(\||{\mathbf{r}^{(k+1)}}-\mathbf{W}\_{\mathbf{r}^{(k+1)}|\mathbf{r}^{(k)}}\mathbf{r}^{(k)}\||\_2^2)+\epsilon_k\||\mathbf{W}\_{\mathbf{r}^{(k+1)}|\mathbf{r}^{(k)}}\||\_F^2}. \end{eqnarray} In this context, $\epsilon_k$ parameter acts as a regularizing coefficient for the linear estimation problem integral to measuring linear dependence between the two arguments of the CMI. Maximizing the CMI given by equation (2) can be accomplished by increasing the correlative entropy of $\mathbf{r}^{(k+1)}$ while decreasing the correlative entropy of the estimation error $\mathbf{e}^{(k+1)}\_\*$. Based on Eq. (A) above, we have $\mathbf{R}\_{\mathbf{r}^{(k+1)}}\succeq \mathbf{R}\_{\mathbf{e}^{(k+1)}\_\*}$, and given that we can write the CMI in (2) as, $$\overset{\rightarrow}{{I}^{(\epsilon_k)}}(\mathbf{r}^{(k)}, \mathbf{r}^{(k+1)})=\frac{1}{2}\sum_{l=1}^{N_{k+1}}(\log(\lambda_l(\mathbf{R}\_{\mathbf{r}^{(k+1)}})+\epsilon_k)-\log(\lambda_l(\mathbf{R}\_{\mathbf{e}^{(k+1)}\_\*})+\epsilon_k))$$ where $\lambda_l(\mathbf{R})$ denotes the $l$-th eigenvalue of the matrix $\mathbf{R}$. We anticipate that the choice of $\epsilon_k$ will primarily influence the correlative entropy of $\mathbf{e}^{(k+1)}\_\*$. Indeed, since $\epsilon_k$ is added to all the eigenvalues of $\mathbf{R}\_{\mathbf{e}^{(k+1)}\_\*}$, reducing its eigenvalues below $\epsilon_k$ would have only incremental increase in the mutual information. As such, a smaller $\epsilon$ value will place greater emphasis on reducing the estimation error $\mathbf{e}^{(k+1)}\_\*$. Consequently, one can consider $\epsilon^{-1}$ can be viewed as an indicator of the sensitivity of the CMI to the levels of estimation error $\mathbf{e}^{(k+1)}\_\*$ (hence act as a conductance for basal/appical-soma connections), determining how far we need to push down the estimation error values to increase the CMI. In brief, the choice of $\epsilon_k$ refers to choice of a CMI from a family of CMIs. Furthermore, the choice of $\epsilon_k$ determines the relative contributions of first and second terms in (2), where smaller $\epsilon_k$ (or larger $\epsilon_k^{-1}$) choice give more emphasis to prediction error entropy. [a] Xiao Fu et. al. Robust volume minimization-based matrix factorization for remote sensing and document clustering. IEEE TSP, Aug 2016. [b] Tatli G et. al. A Bayesian Perspective for Determinant Minimization Based Robust Structured Matrix Factorization. In ICASSP 2023 Jun 2023.
Response to Reviewer Comment - Part 2
In the light of these explanations, we can provide following answers to your questions: >It seems like the answer regarding the terms is that they are actually relatively large, not small. I suppose this is a valid choice, although I'd be curious to know whether the quadratic term in the Taylor expansion is actually negligible compared to the linear one that you keep. The use of a Taylor series-based approximation allows us to transform the nonlinear function of error covariance into an affine form. This transformation facilitates a more convenient network implementation. We acknowledge that there might be inaccuracies in this approximation with potentially a high margin of error, especially depending on the evolution of prediction error correlation matrix. The direct use of the error entropy or alternative approximation strategies might be considered as potential future extensions of our framework. >Regardless of this, such a large value for goes well beyond regularization in the sense of avoiding divergences, it appears to act more like a prior, pushing your system in a direction that presumably is desirable for some reason. What is that direction and why do you want to bias the system like this? As explained above, the choice of $\epsilon_k$ essentially determines the relative contributions of embedding and prediction error correlation matrices on the CMI. Lower $\epsilon_k$ implies giving more emphasis to prediction error related component of the CMI. This fact is more evident in the linear approximation based version of the CMI: $$\overset{\rightarrow}{{I}^{(\epsilon_k)}}(\mathbf{r}^{(k)}, \mathbf{r}^{(k+1)}) \approx \frac{1}{2} \log \det \left(\mathbf{R}\_{\mathbf{r}^{(k+1)}}\right)- \frac{\epsilon_k^{-1}}{2} \text{Tr} \left(\mathbf{R}\_{\overset{\rightarrow}{\mathbf{e}^{(k+1)}\_\*}} \right)+const$$ where $\epsilon_k^{-1}$ is the weight of prediction error covariance dependent term. This is also linked to the fact that $\epsilon_k^{-1}$ acts as a conductance term for incorporating forward and backward prediction errors. > In other words, by using a large $\epsilon$ , you are not quite optimizing correlative mutual information anymore, and in that case, you should explain what you are optimizing instead, and why. Regardless of the choice for $\epsilon_k$, the associated $\overset{\rightarrow}{{I}^{(\epsilon_k)}}(\mathbf{r}^{(k)}, \mathbf{r}^{(k+1)})$ is a legitimate CMI. Thus, we are optimizing a CMI selection from a set of CMIs. The choice of $\epsilon_k$, and therefore, the choice of the corresponding CMI, determine the relative influence of the correlation matrices of the embeddings and prediction errors. It is important to note that these $\epsilon_k$ values are neural network hyperparameters and can be adjusted for better accuracy of the network.
Thanks for the explanations. I think at this point it's just a question of semantics: sure, you can think of each value of $\epsilon$ as defining a new CMI, but the larger that value, the farther that quantity becomes from the concept of mutual information that's it's based on and named after. I think that's fine as long as this is clearly explained in the text. I also think it takes away a bit of the claim of a purely normative approach, but it's not much.
Response to Reviewer Comment
We deeply appreciate your comment and recognize the nuances of the topic at hand. To ensure there's no ambiguity, please allow us to offer further clarity on this matter: **TLDR:** You've correctly pointed out that as $\epsilon$ increases , the CMI-${{I}^{(\epsilon)}}(\mathbf{x}, \mathbf{y})$- deviates from the form of Shannon Mutual Information associated with two jointly Gaussian vectors. However, it is still a legitimate *correlative* mutual information measure that reflect correlation (not necessarily dependence) between two random vectors. We draw a comparison between Shannon Mutual Information and Correlative Mutual Information to clarify our stance: **Shannon Mutual Information** Shannon Mutual Information (SMI), $I_{SMI}(\mathbf{x},\mathbf{y})$ between two random vectors $\mathbf{x}$ and $\mathbf{y}$, we can list the following properties: - Its computation typically requires full joint pdf $f_{\mathbf{x},\mathbf{y}}$, - It is nonnegative, i.e., $I_{SMI}(\mathbf{x},\mathbf{y})\ge 0$, - It is zero,i.e., $I_{SMI}(\mathbf{x},\mathbf{y})=0$ if and only if $\mathbf{x}$ and $\mathbf{y}$ are **independent**, - Due to the previous property, $I_{SMI}$ is a measure of **dependence** between two vectors, - If $\mathbf{x}$ and $\mathbf{y}$ are zero-mean jointly Gaussian non degenerate vectors, i.e., $\left[\begin{array}{c}\mathbf{x}\\\ \mathbf{y}\end{array}\right]\sim \mathcal{N}\left(\mathbf{0},\left[\begin{array}{cc} \mathbf{R}\_\mathbf{x} & \mathbf{R}\_{\mathbf{x}\mathbf{y}} \\\ \mathbf{R}\_{\mathbf{y}\mathbf{x}} &\mathbf{R}\_\mathbf{y}\end{array}\right]\right)$, then $$I_{SMI}(\mathbf{x},\mathbf{y})=\frac{1}{2}\log\det(\mathbf{R}\_\mathbf{x})-\frac{1}{2}\log\det(\mathbf{R}\_{\mathbf{e}\_{\mathbf{x}|\mathbf{y}}}),$$ where $\mathbf{R}\_{\mathbf{e}\_{\mathbf{x}|\mathbf{y}}}=\mathbf{R}\_\mathbf{x}-\mathbf{R}\_{\mathbf{x}\mathbf{y}}\mathbf{R}\_{\mathbf{y}}^{-1}\mathbf{R}\_{\mathbf{y}\mathbf{x}}$ is the error correlation matrix of the best linear minimum mean square (MMSE) estimator of $\mathbf{x}$ from $\mathbf{y}$. **Correlative Mutual Information** Correlative Mutual Information (CMI) with parameter $\epsilon \ge 0$, $I^{(\epsilon)}(\mathbf{x},\mathbf{y})$ between two random vectors $\mathbf{x}$ and $\mathbf{y}$, which is defined as $$I^{(\epsilon)}(\mathbf{x},\mathbf{y})=\frac{1}{2}\log\det(\mathbf{R}\_\mathbf{x}+\epsilon\mathbf{I})-\frac{1}{2}\log\det(\mathbf{R}\_{\mathbf{e}\_{\mathbf{x}|\mathbf{y}}}+\epsilon\mathbf{I}),$$ where $\mathbf{R}\_{\mathbf{e}\_{\mathbf{x}|\mathbf{y}}}=\mathbf{R}\_\mathbf{x}-\mathbf{R}\_{\mathbf{x}\mathbf{y}}(\mathbf{R}\_{\mathbf{y}}+\epsilon \mathbf{I})^{-1}\mathbf{R}\_{\mathbf{y}\mathbf{x}}$ is the error correlation matrix of the best $\epsilon$-regularized linear MMSE estimator of $\mathbf{x}$ from $\mathbf{y}$. we can list the following properties for the CMI: - Its computation only requires second order statistics, i.e., $\mathbf{R}\_\mathbf{x}$, $\mathbf{R}\_{\mathbf{x}\mathbf{y}}$ and $\mathbf{R}\_\mathbf{y}$, irrespective of the joint pdf $f_{\mathbf{x}\mathbf{y}}$, - It is always nonnegative, i.e., $I^{(\epsilon)}(\mathbf{x},\mathbf{y})\ge 0$. - It is zero,i.e., $I^{(\epsilon)}(\mathbf{x},\mathbf{y})=0$ if and only if $\mathbf{x}$ and $\mathbf{y}$ are **uncorrelated**, - Due to the previous property, $I^{(\epsilon)}$ is a measure of **correlation** between two vectors, - For the choice $\epsilon=0$, $I^{(0)}(\mathbf{x},\mathbf{y})$ is equal to $I_{SMI}$ of two jointly Gaussian vectors with the same second order statistics. As a result, for the CMI $I^{(\epsilon)}(\mathbf{x},\mathbf{y})$ with a choice of parameter $\epsilon>0$ (stricly positive), we can make the following observations: - ${I}^{(\epsilon)}$ is not equal to the form of $I_{SMI}$ betweeen two Gaussian vectors, - Therefore, it is not a measure of dependence, - However, it is still a legitimate measure of correlation between two random vectors, i.e., $I^{(\epsilon)}(\mathbf{x},\mathbf{y})\ge 0$ with equality if and only if $\mathbf{x}$ and $\mathbf{y}$ are uncorrelated (Lemma 1 of [35]).
If I understand correctly, the argument is that we are more interested in the fact that the CMI is a measure of correlation between two vectors. But there are many possible measures of correlation—simplest one being the correlation itself. Maximizing correlation leads to CCA, as for example Lishutz, Bahroun, Golkar et al. did in "A Biologically Plausible Neural Network for Multichannel Canonical Correlation Analysis", *Neural Computation* (2021). How different is your approach compared to (a nonlinear version of) that? Of course, I'm not suggesting you should change your approach at this point! I'm just saying that there were many choices made along the way that led to your method as opposed to something else, and I think these should be acknowledged better.
Response to Reviewer Comment
Thank you for your insightful comment and request for clarification. Let's delve deeper into the relationship between Canonical Correlation Analysis (CCA) and Correlative Mutual Information (CMI), as well as how reference [a] relates to our framework: **Canonical Correlation Analysis** Given two zero-mean random vectors $\mathbf{x}\in \mathbb{R}^m$ and $\mathbf{y}\in \mathbb{R}^n$, CCA finds projection matrices $\mathbf{V}\_\mathbf{x}\in \mathbb{R}^{p \times m}$ and $\mathbf{V}\_\mathbf{y}\in \mathbb{R}^{p \times n}$, where $p=\text{min}(m,n)$, such that the pairwise correlations of the elements of projection vectors $\mathbf{q}=\mathbf{V}\_\mathbf{x}\mathbf{x}$ and $\mathbf{w}=\mathbf{V}\_\mathbf{y}\mathbf{y}$ are maximized. Specifically, the aim is to maximize the sum of the correlations $\rho_i=E(q_iw_i)$, while maintaining unity-variance output constraints on the projection matrices. **Connection between CCA and CMI** As per reference [b], when $\mathbf{x}$ and $\mathbf{y}$ are jointly Gaussian, Shannon Mutual Information of $\mathbf{x},\mathbf{y}$ (and consequently the CMI with parameter $\epsilon=0$) can be expressed in terms of cannonical correlations as: $$ I^{(0)}(\mathbf{x},\mathbf{y})=-\frac{1}{2}\sum_{i=1}^p\log(1-\rho_i^2)\hspace{0.2in} (*)$$ Thus, while CCA can be employed to compute $I^{(0)}(\mathbf{x},\mathbf{y})$, it doesn't inherently maximize it. **About the biologically plausible CCA approach of [a]** In essence, reference [a] introduces a framework for biologically plausible neural networks that produce projections $\mathbf{q}$ and $\mathbf{w}$ with optimal canonical correlations from inputs $\mathbf{x}$ and $\mathbf{y}$. For the biologically plausible CCA network of [a], we can list the following: - the samples of $\mathbf{x}$ and $\mathbf{y}$ serve as strict inputs; they are not manipulated by the CCA network, - the network computes the projections $\mathbf{q}$ and $\mathbf{w}$ of these inputs, which have maximum sum of elementwise correlations. - Overall the network calculates CCA components corresponding to its inputs as a result of a dynamical optimization process. However, this network does not manipulate $\mathbf{x}$ and $\mathbf{y}$ to maximize their canonical correlations, - The element-wise correlations of the outputs of the CCA network correspond to canonical correlations $\rho_i$, which can then be used to determine the CMI $I^{(0)}(\mathbf{x},\mathbf{y})$. **The relationship of CorInfoMax framework and the reference [a]** As described above, - the biologically plausible CCA networks of Reference [a] can be used to calculate CCA components corresponding the inputs $\mathbf{x}$ and $\mathbf{y}$. As such, it can also be employed to compute the CMI $I^{(0)}(\mathbf{x},\mathbf{y})$ using Eq. (*) above if extended. However, the CCA network in [a] doesn't inherently aim to maximize the canonical correlations by manipulating its inputs. - In contrast, CorInfoMax networks strive to optimize the CMIs between network layers, i.e., $I^{(\epsilon)}(\mathbf{r}^{(k)},\mathbf{r}^{(k+1)})$ for $k=0, \ldots, P-1$. Our multi-layered, sample-based CorInfoMax objective (as defined in Equation (9) in our paper) facilitates the evolution of hidden layer activities and the output vector $\mathbf{y}$ to maximize the flow of correlative information across consecutive layers. - An intriguing future extension might involve leveraging the CCA networks from [a] as subnetworks within a CMI maximization context, where the CMI objective in (9) modified based on Eq. (*) above. As an important note, among different correlation measures, the correlative entropy and the corresponding correlative mutual information possess useful properties of being second-order-statistics based uncertainty and information measures (such as additivity of self information for uncorrelated vectors). These properties are particularly useful for obtaining clear views for interpretibility. Given the relevance of the correlation concept and biological networks, we'll incorporate [a] into our references. We're grateful for your guidance on this matter. [a] Lipshutz D, Bahroun Y, Golkar S, Sengupta AM, Chklovskii DB. A biologically plausible neural network for multichannel canonical correlation analysis. Neural Computation. 2021 Aug 19;33(9):2309-52. [b] Bach, Francis R., and Michael I. Jordan. "Kernel independent component analysis." Journal of machine learning research 3.Jul (2002): 1-48.
(There's a small typo in (*): the product should have been a sum.) I'm confused about the claim that CCA does not maximize the mutual information for Gaussian variables. Given that the log is an increasing function, maximizing (*) should certainly find the top canonical correlation when $p = 1$ (a similar claim is made in "Canonical correlation analysis based on information theory", *Journal of multivariate analysis* (2004). (https://core.ac.uk/download/pdf/82657228.pdf). Is the claim that there is a problem when generalizing this to $p > 1$? I don't really see how this could fail.
Response to Reviewer Comment
Firstly, thank you for pointing out the typo. We've corrected it in our previous response. We apologize for any confusion our prior communication might have caused. To provide further clarity on the topic: - In our earlier response, we provided the expression for the CMI $I^{(0)}(\mathbf{x},\mathbf{y})$ using cannonical correlations $\rho_1, \ldots, \rho_p$: $$ I^{(0)}(\mathbf{x},\mathbf{y})=-\frac{1}{2}\sum_{i=1}^p\log(1-\rho_i^2)\hspace{0.2in} (*)$$ - Thus, maximizing (*) by adjusting the second order statistics of $\mathbf{x}$ and $\mathbf{y}$, i.e., ($\mathbf{R}\_{\mathbf{x}}$, $\mathbf{R}\_{\mathbf{y}}$, $\mathbf{R}\_{\mathbf{x}\mathbf{y}}$) would correspond to correlative information maximization. If we were to clarify CCA versus CorInfoMax, these are two different maximizations over different optimization variables: - **CCA**: With the given second-order statistics ($\mathbf{R}\_{\mathbf{x}}$, $\mathbf{R}\_{\mathbf{y}}$, $\mathbf{R}\_{\mathbf{x}\mathbf{y}}$), the goal is to determine the canonical coordinates $\mathbf{q}=\mathbf{V}\_\mathbf{x}\mathbf{x}$ and $\mathbf{w}=\mathbf{V}\_\mathbf{y}\mathbf{y}$, and the corresponding canonical correlations through the optimization: $$ \text{maximize }\_{\mathbf{V}\_\mathbf{x},\mathbf{V}\_\mathbf{y}} \text{Tr}(\mathbf{V}\_\mathbf{x}\mathbf{R}\_{\mathbf{x}\mathbf{y}}\mathbf{V}\_\mathbf{y}^T) \hspace{0.1in} (**)$$ $$ \text{subject to } \mathbf{V}\_\mathbf{x}\mathbf{R}\_\mathbf{x}\mathbf{V}\_\mathbf{x}^T+\mathbf{V}\_\mathbf{y}\mathbf{R}\_\mathbf{y}\mathbf{V}\_\mathbf{y}^T=\mathbf{I}.$$ Here, the optimization variables are $\mathbf{V}\_\mathbf{x}\in \mathbb{R}^{p \times m}$ and $\mathbf{V}\_\mathbf{y}\in \mathbb{R}^{p \times n}$. The second order statistics ($\mathbf{R}\_{\mathbf{x}}$, $\mathbf{R}\_{\mathbf{y}}$, $\mathbf{R}\_{\mathbf{x}\mathbf{y}}$) are not optimization variables, but they serve as input parameters to this optimization. - **Correlative Information Maximization**: This involves maximizing $I^{(\epsilon)}(\mathbf{x},\mathbf{y})$ by adjusting the second order statistics ($\mathbf{R}\_{\mathbf{x}}$, $\mathbf{R}\_{\mathbf{y}}$, $\mathbf{R}\_{\mathbf{x}\mathbf{y}}$) subject to constraints on $\mathbf{x},\mathbf{y}$ (such as the set membership constraints) In summary, CCA optimization variables are projection matrices, $\mathbf{V}\_\mathbf{x}\in \mathbb{R}^{p \times m}$ and $\mathbf{V}\_\mathbf{y}\in \mathbb{R}^{p \times n}$, while for CorInfoMax optimization variables are the second order statistics ($\mathbf{R}\_{\mathbf{x}}$, $\mathbf{R}\_{\mathbf{y}}$, $\mathbf{R}\_{\mathbf{x}\mathbf{y}}$) - Our claim in our previous response was that the biological CCA approach in reference [a] achieves the solution for the online version of the optimization (**). It doesn't alter the input statistics ($\mathbf{R}\_{\mathbf{x}}$, $\mathbf{R}\_{\mathbf{y}}$, $\mathbf{R}\_{\mathbf{x}\mathbf{y}}$). - In other words, the CCA network in [a] implements this maximization over the projection matrices to find canonical coordinates/canonical correlations corresponding to the provided input statistics. However, there is no maximization of the corresponding CMI by manipulating the inputs (and their statistics ($\mathbf{R}\_{\mathbf{x}}$, $\mathbf{R}\_{\mathbf{y}}$, $\mathbf{R}\_{\mathbf{x}\mathbf{y}}$)). - In the correlative information maximization process, we actually manipulate the inputs $\mathbf{x}$ and $\mathbf{y}$ ( which are layer activations $\mathbf{r}^{(k)}$ and $\mathbf{r^{(k+1)}}$ in our framework) to manipulate their statistics to maximize the CMI $I^{(\epsilon)}(\mathbf{x},\mathbf{y})$, under the set membership constraints on layer activations. - As we proposed in our previous response, a CorInfoMax extension could involve CCA networks extracting the canonical correlations vector $\rho$ for current statistics, while an outer optimization maximizes (*) by adjusting the statistics. We would be happy to provide further clarifications if needed. [a] Lipshutz D, Bahroun Y, Golkar S, Sengupta AM, Chklovskii DB. A biologically plausible neural network for multichannel canonical correlation analysis. Neural Computation. 2021 Aug 19;33(9):2309-52.
Thanks for the further clarifications!
Response to the Reviewer Comment
We would like to thank you once again for your valuable insights and thoughtful inquiries. Your considerate observations have helped us to enhance and clarify our methodology, and we are sincerely grateful for your overall feedback.
Thank you to the authors for their detailed response and explanations. Regarding the additional experiments, may I ask why feedback alignment with cross entropy loss is not included in the experiments presented in the new Tables 1/2 (whilst cross entropy loss is used with standard BP)?
Response to Reviewer Comment
Thank you again for your comments and questions. We considered MSE loss appropriate for the feedback-alignment experiments, especially since we employ MSE loss within the CorInfoMax framework. However, we will be happy to incorporate feedback-alignment experiments utilizing cross-entropy loss. We have initiated these experiments and will share the results once they are available.
Response to Reviewer Comment - Continued
We have completed our experiments for Feedback Alignment with the Cross-Entropy loss. Below, you can find the updated results in Table 1 for 2-layer networks and Table 2 for 3-layer networks. **Table 1: 2-layered Neural Network Results** Algorithm \ Dataset | MNIST | Fashion-MNIST | CIFAR10 :--------------------:|:--------------------------------:|:--------------------:|:-------------------------: CorInfoMax-$\mathcal{B}_{\infty, +}$ | 97.62 $\pm$ 0.1 | 88.14 $\pm$ 0.3 | 51.86 $\pm$ 0.3| CorInfoMax-$\mathcal{B}_{1, +}$ | 97.71 $\pm$ 0.1 | 88.09 $\pm$ 0.1 | 51.19 $\pm$ 0.4| EP | 97.61 $\pm$ 0.1| 88.06 $\pm$ 0.7 | 49.28 $\pm$ 0.5 | CSM | 98.08 $\pm$ 0.1| 88.73 $\pm$ 0.2 | 40.79 | PC | 98.17 $\pm$ 0.2 | 89.31 $\pm$ 0.4 | - | PC-Nudge | 97.71 $\pm$ 0.1| 88.49 $\pm$ 0.3 | 48.58 $\pm$ 0.7| Feedback-Alignment (with MSE) | 97.99 $\pm$ 0.03| 88.72 $\pm$ 0.5| 50.75 $\pm$ 0.4 | Feedback-Alignment (with CrossEntropy)| 97.95 $\pm$ 0.08 | 87.38 $\pm$ 0.92 | 52.37 $\pm$ 0.38 | BP (with MSE) |97.58 $\pm$ 0.01 |88.39 $\pm$ 0.1 |52.75 $\pm$ 0.1 | BP (with CrossEntropy) | 98.27 $\pm$ 0.03| 89.41 $\pm$ 0.2| 53.96 $\pm$ 0.3 | **Table 2: 3-layered Neural Network Results** Algorithm \ Dataset | MNIST | CIFAR10 | CIFAR100 | :--------------------:|:-------------------------:|:--------------------:|:---------------------------| CorInfoMax-$\mathcal{B}_{\infty, +}$ | 97.58 $\pm$ 0.1 | 50.97 $\pm$ 0.4 | 20.84 $\pm$ 0.4 | Feedback-Alignment (with MSE) | 98.18 $\pm$ 0.0 | 50.26 $\pm$ 1.4| - | Feedback-Alignment (with CrossEntropy)| 97.96 $\pm$ 0.15 | 51.64 $\pm$ 0.60 | - | BP (with MSE) | 97.74 $\pm$ 0.0 | 55.49 $\pm$ 0.04 | 26.56 $\pm$ 0.2 | BP (with CrossEntropy) | 98.28 $\pm$ 0.04 | 56.14 $\pm$ 0.3 | 28.93 $\pm$ 0.3 |
Thank you to the authors for their new simulations. On one hand I am a bit disappointed to see that CorInfoMax does not improve upon standard FA for multi-layer networks, but on the other hand the authors have convinced me that the work might still be a very valuable contribution for neuroscience and bio-plausible learning. I also appreciate the great deal of work this study must have taken and the authors' commitment to rigorous explanations in their rebuttal response. I will upgrade my score by 1.
Decision
Accept (poster)