Unconstrained Stochastic CCA: Unifying Multiview and Self-Supervised Learning

Thank you for such a thorough and insightful review. We appreciate your positive feedback with regard to our paper's soundness, clear layout, and relevance to the community. Thank you in particular for your constructive criticism. We found this extremely helpful and propose a number of improvements to the text in response. In the following comment, we address each of your main comments in detail, outlining improvements where relevant. We sincerely hope these resolve any concerns you may have. If so, we kindly ask that you consider raising your score. Regardless of the outcome, we thank you again for such helpful feedback. ### Minor and typos - Yes, assuming the observations are from a continuous probability distribution is sufficient to give linear independence with probability 1. We have modified the footnote accordingly. - References: we have fixed the capitalisation and will seek advice from the area chair regarding the long author lists.

### Lemma 3.2 - equivalence to previous Deep CCA formulations Thank you for raising Lemma 3.2. On reflection, we agree that the proof was unclear - and have taken the opportunity to rewrite it with more careful notation. Unfortunately, this took us over the 9 page limit, so we only give a sketched summary in the main text and give full details appendix C.1 (referenced in the main-text sketched summary). To directly answer your first question: the parameter $\theta$ relates to the whole neural network. To better explain the local-optimum argument we introduce notation: decompose $\theta = (U, \phi)$ into a set of final-layer weights $U$, and a set of weights for the previous layers $\phi$. Then if $\hat{\theta} = (\hat{U}, \hat{\phi})$ is a local optimum of $\mathcal{L}(\theta)$, then $\hat{U}$ is a local optimum of the function $l(U) = \mathcal{L}((U,\hat{\phi}))$. Though in practice we would not reach a local optimum in finite time with SGD, one may expect the final estimates to be near a local optimum, and therefore for the conclusions of the lemma to apply at least approximately. Does that answer your question? (and are you satisfied with our proposed change? or is there more we can do?) ### Discussion on Barlow Twins and VICReg Thank you for raising section 3.4. On reflection, we agree that (what was) Proposition 1 could be better integrated with the main text, and that it may be confusing to readers who do not know about Barlow twins and VICReg. We have therefore taken the opportunity to rewrite the section. Our proposals include: - Focusing on VICReg - and writing the VICReg loss explicitly. (there was not enough space to state both losses, and we have more complete theoretical analysis for VICReg than Barlow twins). - Replacing the 'informal' proposition with a sequence of bullet point conclusions. - Brief sentences to better motivate why these results are of interest to the SSL community. Do you agree that these changes improve the presentation? (and is there more we can do to improve this section?) ### Uniqueness for CCA Your question about uniqueness is a good one - and a detail that we had hoped to 'brush under the carpet' (we wanted to focus on computation, and not get bogged down in regularisation). We will give a very short answer here due to space constraints, but hope to expand upon this later in the discussion. For GEPs with $B$ strictly positive definite there is non-uniqueness up to choice of basis within each generalised eigenspace. In high dimensional *sample* CCA there is extreme degeneracy - all canonical correlations are 1 (as discussed in the main text) and there is additional degeneracy in the canonical directions due to the low-rank structure of the sample covariances (the full-batch $\hat{B}$ has non-trivial kernel). This degeneracy is overcome by sample ridge CCA by enforcing that $\hat{B}_\alpha$ is strictly positive definite (note however, that the generalised eigenvalues no longer correspond to canonical correlations, and need not be in [0,1], complicating interpretation). This phenomenon is analogous to the phenomenon that ridge regression will find unique solutions even in underdetermined systems where OLS gives a degenerate family of solutions (which we suspect motivated your question). Does this clarify matters? ### Clarification on the UK Biobank Task We have also made some small improvements in response to your question about the Partial Least Squares task - thank you for raising this. - *features:* the UK BioBank consists of real-valued continuous brain volumes and ordinal, integer genetic variants. We have added this information to appendix I.3.4. - *modelling and "convenience" considerations:* could you please clarify what you are referring to here - we do not believe we used the word "convenience" in this UK Biobank context. - *precise task:* it is now standard in Imaging-Genetics research to perform (full sample) CCA or PLS on multi-view datasets, in order to interpret the estimated directions or to use transformed variables for downstream tasks. We use stochastic PLS in this scenario because the dataset is extremely high dimensional and has far more features than samples ($D \gg N$) so un-regularised CCA would break down - moreover, it is more convenient to run a single PLS than to perform tuning parameter selection for ridge CCA. With regard to equations 4, 5, 6, 7: we (hoped to) define PLS through our 'Unified GEP formulation' (equation 7) with $\alpha_i = 1 \, \forall i$, and here there are two views ($I=2$). In this case equation 7 specialises to give the GEP defined by $$A = \begin{pmatrix} 0 &\operatorname{Cov}(X^{(1)}, X^{(2)}) \\\\ \operatorname{Cov}(X^{(2)}, X^{(2)}) & 0 \end{pmatrix}, \quad B = \begin{pmatrix}I_{D_1} & 0 \\\\ 0 & I_{D_2} \end{pmatrix}, \quad u =\begin{pmatrix} u^{(1)} \\\\ u^{(2)} \end{pmatrix}.$$ We have also added this equation to appendix I.3.4, with a short discussion. Does that answer your question?

Thank you once again for your careful review and helpful feedback. We admit that we had not planned to expand upon the CCA-uniqueness discussion in the paper; but we agree that it would be good to include more details. We will consider how to add appropriate discussion to our Appendix C on CCA background for the camera-ready version. Thank you also for clarifying your comment regarding the UK Biobank dataset. To attempt to answer your questions: 1. **Modelling:** This raises some difficult issues, of a more philosophical nature. Our subjective opinion is that there is not a single statistical model associated with CCA, ridge CCA or PLS, and indeed the methods can be applied without reference to a probabilistic model. That being said, CCA, ridge CCA and PLS can (all) be interpreted as linear Gaussian models - and indeed there are a number of slightly different probabilistic formulations in the literature already, e.g. [1,2,3,4] (incidentally, we hope to unify these in future work). These Gaussian models may not be perfectly 'realistic' but they are often reasonable approximations to reality. Finally, to redirect this discussion, we would like to reiterate that the main reason we worked on the CCA, ridge-CCA, PLS family is because these methods are already very popular in practice - both due to their simplicity and empirically-established usefulness. 2. **Tractability:** Yes, our algorithm will run in a reasonable amount of time without linear algebra errors (there are no inverses, simply matrix multiplications) - even for unregularised high-dimensional full-sample CCA where the solutions are degenerate. (Interestingly, our algorithm appears to enjoy a regularising effect from SGD in this degenerate case, and often learns solutions with high out-of-sample correlations; but we had to leave further exploration of this phenomenon for later work.) To summarise our motivation for the experiment: we wanted to illustrate the computational benefits of our methods on real, high-dimensional UK Biobank data; we had a choice of using any method on the ridge-regularised spectrum between CCA and PLS. Un-regularised full-sample CCA is degenerate, and, though a general ridge CCA would be a reasonable in principle, it could require careful tuning parameter selection (and associated discussion). By focusing on PLS we could avoid tuning parameter selection entirely. An added advantage of PLS is that it is particularly popular in similar biomedical contexts, easing comparison with previous work. We admit that it would also be interesting to compare our PLS analysis to appropriately tuned ridge CCA, and hope to investigate this further. We realise this response does not closely relate to your original comment about the UK Biobank task, and contains certain subjective opinions, but we hope that it helps clarify the points raised in your most immediate comment. Please let us know if there is anything we can improve upon or clarify further. Thank you again for your help. [1] M G Gustafsson, "A probabilistic derivation of the partial least-squares algorithm", J Chem Inf Comput Sci.  2001 Mar-Apr;41(2):288-94. [2] De Bie & De Moor, "On the Regularization of Canonical Correlation Analysis", ICA 2003, https://www.researchgate.net/publication/229057909_On_the_Regularization_of_Canonical_Correlation_Analysis [3] Bach & Jordan, "A Probabilistic Interpretation of Canonical Correlation Analysis", UC Berkeley technical report, 2005 [4] Said el Bouhaddani, et al. "Probabilistic partial least squares model: Identifiability, estimation and application." arXiv preprint arXiv:2010.00554 (2020).

Thank you for agreeing that the results of our extensive experiments prove the effectiveness of our proposed method. We appreciate this opportunity to address the concerns raised in your review, particularly regarding the novelty of our approach compared to prior work such as [1]. We have addressed your concern about our decision not to perform PCA preprocessing in our common response. We hope that you find this helpful. In the remainder of this response, we will reiterate the novel aspects of our proposed method, with particular emphasis on comparisons to Gemp et al. (2022) [1], as you requested. Comparing directly to [1] we will argue that our work is of much broader scope and that even in the narrow scope where our work is comparable to [1], our method is very different (and in fact has a number of advantages, which we hope to convince you of). We will also argue that our work is novel within the literature more broadly. By elaborating on these points, we hope to persuade you of the novelty of our contribution. We sincerely hope that this response successfully addresses your concerns; if so we kindly ask that you consider raising your score. Regardless of the outcome, we thank you again for the time you have taken to review our work. ## Novelty of our Proposed Methods Firstly, we address novelty more broadly. Our Eckhart-Young inspired characterisation for GEPs is novel - it does not appear elsewhere in the literature (to our knowledge). Our presentation of a single framework and class of algorithms to tackle stochastic CCA, PLS and Deep CCA is also novel, as is the application of Deep CCA to SSL. Even our stochastic GEP algorithm is very different to all previous proposals - it is the first such algorithm formulated with an explicit loss function that permits unbiased gradient updates. The work of Gemp et al. (2022) is narrower in that it only proposes algorithms for stochastic GEPs. Their algorithms do not have an explicit loss function, and instead use update steps which are inspired by their previous work on an 'Eigengame' for PCA (a very different motivation). Crucially, this means that their method cannot be extended to Deep CCA, let alone SSL; these deep extensions were a key focus of our work, and comprised the majority of our experiments (3 out of 5). Even if we only compare proposals for stochastic GEPs, the $\gamma$-eigengame method of Gemp et al. [1] is very different to our method, and is in fact problematic in a number of ways. We reiterate that our method is derived by applying SGD to a population loss function that characterises GEPs, has unbiased update steps and only involves a single learning rate parameter for SGD. By contrast $\gamma$-eigengame (Algorithm 2 in [1]): - **Does not have an explicit loss function.** Instead motivates update directions by a heuristic argument. - **Has biased update steps,** because it involves normalisation by sample quantities. - **Has flimsier theoretical support:** they use an error-propagation argument to justify their claim that their algorithm converges with probability 1. However, this proof appears incomplete (red flags include "Assume $\epsilon_i'$ is $\mathcal{O}(\epsilon_i)$ to ease the exposition"). By contrast, we can directly apply results from the literature on SGD to deduce convergence with probability 1. - **Most importantly, their method is much more complicated.** It requires "introducing an auxiliary variable, denoted $[B\hat{v}]_j$ to track the running averages of $B\hat{v}_j$" and an additional step to "manually clip the result to be greater than or equal to $\rho$, the minimum singular value of $B$". Their method therefore requires knowledge of a lower bound $\rho$ to the singular values of $B$, which is hard to justify in practice. Also, they require an additional learning rate parameter / step size sequence $\gamma_t$ for updates of $[B\hat{v}]_j$. The performance of their method depends delicately on the choice of this parameter, and tuning it induces a significant computational overhead. Finally, we highlight the practical advantages of our method over $\gamma$-eigengame, as illustrated in our experiments. If anything, our proposal is most closely related to the update rule of SGHA [2]; yet because this cannot be written as the gradient of a population loss function it cannot be easily extended to Deep CCA, and again has flimsier theoretical support. [1] Gemp, Ian, Charlie Chen, and Brian McWilliams. "The generalized eigenvalue problem as a Nash equilibrium." arXiv preprint arXiv:2206.04993 (2022). [2] Chen, Zhehui, et al. "On constrained nonconvex stochastic optimization: A case study for generalized eigenvalue decomposition." The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019.

We sincerely appreciate your positive feedback on our work. Thank you for acknowledging its originality, quality, clarity, and significance, especially in the context of CCA-type problems and self-supervised learning methods. Thank you also for your constructive criticism. We have corrected the typo in section 3.3. We agree with you that it would be valuable to have a clearer comparison of properties of the presented methods with the other methods in our benchmarks. You are right in that it is not straightforward to decide what information is most relevant to display. We are still working on this, and hope to include such a summary in the appendix shortly. We hope that our detailed responses to the other reviews will reinforce your confidence in our work.