**[Amari Distance Computation]**
In our evaluation, we used $\bf{\hat{\Theta}^{(q)}} = \bf{Q}^{(q)} \bf{A}^{(q)}$$(1:d\_C)$, where $\bf{A}^{(q)}$$(1:d\_C)$ represents the first $d\_C$ columns of $\bf{A}^{(q)}$. Note that $\bf{Q}^{(q)}$ is our estimated linear operator, and $\bf{A}^{(q)}$ is the ground-truth mixing system that is available for synthetic data experiments (we only evaluated Amari distance for the synthetic data on our previous reply).
Another note is that (thanks to the above discussion) we realized that general matrix distances (such as Euclidean distance) could be a better fit for our case than the Amari distance. To see, recall that we have content identifiability if and only if
$ \bf{Q}^{(q)} \bf{A}^{(q)} = [\bf{\Theta}, \bf{0}]$. Hence, we need $\hat{\bf{\Theta}}^{(1)} = \hat{\bf{\Theta}}^{(2)} = \bf{\Theta}$. However, Amari distance is insensitive (invariant) to permutation and scaling, i.e., $\hat{\bf{\Theta} }^{(1)} = \bf{P} \Lambda \hat{\Theta}^{(2)}$ incurs zero Amari distance, where $\bf{P}$ and $\Lambda$ are any permutation and scaling matrices respectively. Hence, we present the Euclidean distance instead of the Amari distance. Additionally, we also report the $\\| \bf{Q}^{(q)} \bf{A}^{(q)}$$(d\_C+1 : d\_C + d^{(q)}\_P ) \\|_F$ which needs to be close to $\bf{0}$ for identifiability.
Table 1: Numerical evaluation of identifiability $\\| \widehat{\bf{\Theta} }^{(1)} (1: d\_C) - \widehat{ \bf{\Theta} }^{(2)}(1:d\_C) \\|_{F} .$
|N | SCA | CCA |
| :- | :- | :- |
|100,000 | 0.009 | 1.368|
|10,000 | 0.007 | 1.544|
|1,000 | 0.003 | 2.206|
|100 | 0.032 | 1.755|
|50 | 0.133 | 1.667|
|20 | 1.462 | 1.522|
Table 2: $1/2 \sum\_{q=1}^2 \\| \widehat{\bf{\Theta}}^{(q)} ( d\_C+1 : d\_C+d^{(q)}\_P) \\|\_{F}$.
|N | SCA | CCA |
| :- | :- | :- |
|100,000 | 0.021 | 0.284|
|10,000 | 0.034 | 0.279|
|1,000 | 0.002 | 0.329|
|100 | 0.043 | 0.368|
|50 | 0.131 | 4.092|
|20 | 0.747 | 0.755|
We will follow the reviewer’s suggestion and add the new experiment (with mean and standard deviation) for the synthetic data experiments in the revised version.
**Q2.** Yes. We will add a footnote about conditional independence in the main paper.
**Q3.** Thanks for the suggestion. We have run the following experiment with unbalanced data.
For the following experiment, the data for two modalities were generated, by sampling the shared component of dimension($D=2$) from VonMises distribution and private components from Gamma and Laplace distributions. The number of samples in the first modality is fixed to 100,000 and the second view ranges from 10,000 to 10 samples.
Table 3: Performance of SCA on imbalance data based on following two metrics,
**metric1** = $\\| \widehat{\bf{\Theta}}^{(1)} (1: d\_C) - \widehat{ \bf{\Theta}}^{(2)}(1: d\_C) \\|\_{F} $,
**metric2** = $1/2 \sum\_{q=1}^2 \\| \widehat{\bf{\Theta}}^{(q)} ( d\_C+1: d\_C+d^{(q)}\_P) \\|\_{F}$.
| \# samples in modality 2 | metric1 | metric2 |
| :- | :- | :- |
|10,000 | 0.008 | 0.025 |
|1,000 | 0.025 | 0.015 |
|100 | 0.091 | 0.087 |
|10 | 1.375 | 0.209 |
We will include the above result (with mean and standard deviation) in the revised version.
**Q4. [Possible Benefit of $Q \geq 2$ domains]**
One foreseeable benefit of having more than two domains is that Assumption 1, when modified for $Q \geq 2$ domains, could be more relaxed. This is because modality variability in general is satisfied if at least two of the total number of domains satisfy the current Assumption 1. Having more modalities can make the chance of Assumption 1 increased. On the other hand, enforcing the distribution matching constraint Eq. (6b) could be more challenging for more than two domains. We will add this discussion in the revised version.
**[Complexity Analysis]**
The short answer is that **both the memory and computational complexities of the proposed method scales linearly with** $d_C$. The per-iteration computational complexity is $O(B d\_C (d^{(1)} + d^{(2)}) )$, where $B$ is the mini-batch size. The per-iteration memory complexity is $O(B d\_C (d^{(1)} + d^{(2)}) )$ as well. The complexities are based on the fact that we use mini-batch based stochastic gradient-type optimizer. We will add a section in the appendix to detail the complexity calculation.