Independent component analysis (ICA) is a fundamental statistical tool used to reveal hidden generative processes from observed data. However, traditional ICA approaches struggle with the rotational invariance inherent in Gaussian distributions, often necessitating the assumption of non-Gaussianity in the underlying sources. This may limit their applicability in broader contexts. To accommodate Gaussian sources, we develop an identifiability theory that relies on second-order statistics without imposing further preconditions on the distribution of sources, by introducing novel assumptions on the connective structure from sources to observed variables. Different from recent work that focuses on potentially restrictive connective structures, our proposed assumption of structural variability is both considerably less restrictive and provably necessary. Furthermore, we propose two estimation methods based on second-order statistics and sparsity constraint. Experimental results are provided to validate our identifiability theory and estimation methods.
Paper
Similar papers
Peer review
Summary
Aiming to address rotational invariance of Gaussian sources, this work first proposed an ICA identifiability theory based on Structural Variability. It then proposed two methods based on sparsity regularization and continuous constrained optimization to estimate the mixing matrix. It also made connections between ICA and causal discovery. It finally showed preliminary results to validate the proposed theory and estimation methods.
Strengths
1. The proposed method could be potentially useful as it claims to identify linear Gaussian sources with a weaker assumption compared to structural sparsity. 2. The notations, theorems and proofs are clear in general. 3. The proofs or explanations in the Appendix are very detailed and helpful. 4. The author(s) conducted experiments to validate their theory and showed the effectiveness of the proposed SparseICA method in two simulated datasets.
Weaknesses
1. As the author(s) pointed out, my main concern is that there is no sufficient empirical evaluation to demonstrate the effectiveness and generalizability of the proposed theory. Current experiments only used simulated data in two settings. How does the proposed theory work on real-world datasets? This work would be more convincing if there were experiments on real-world datasets. 2. Additionally, the author(s) argued that the theory proposed by Zheng et al. 2022 couldn't capture hierarchical structures (lines 39 - 41), but it doesn't seem that the authors performed experiments to identify Gaussian sources which are organized in a hierarchical manner. 3. I don't feel all theorems/assumptions/examples were explained very clearly, but I also note that the author(s) provided detailed proofs or explanations for each proposed theorem/assumption/example in the Appendix. So I would suggest that the author(s) try to organize the manuscript more logically and guide readers to the corresponding Appendix section in the main text. 4. There are a lot of typos in the present manuscript. For example, line 71: $j$-th column by $a_{:, j}$; line 104: Section 3.2; line 248: there exists; line 254: repeated "with". Please proof-read the manuscript prior to submission. 5. There is no statistical test to compare results. 6. There is no code provided to replicate the results. Please consider making the code publicly available.
Questions
1. Example 1: I understand the derivation in this example but I wonder how to identify such constraints. Do we have to derive these hard or non-hard constraints for each matrix? Are there any general principles to derive such constraints? If so, please briefly explain the principles. If not, it doesn't seem to be computationally efficient to derive these constraints as the dimension of A increases. 2. Line 130: Please elaborate what "sufficiently diverse effect" means exactly. 3. Do you have any explanations about the performance difference from two estimation methods in Figure 1? For example, decomposition method seems worse than likelihood method in vanilla case.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
2 fair
Limitations
See Weaknesses.
Summary
This paper provides theorems under which the mixing matrix of linear ICA with Gaussian sources can be identified. While Gaussian sources are known to be unidentifiable in the classical ica theory, identifiability is possible if certain sparse structure is assumed for the mixing matrix as was initailly shown in Zheng. The assumptions in Zheng were restrictive however. This work expands the work of Zheng to provide more general conditions, namely: - assumption of structural provided is much less restrictive on the sparsity pattern - necessity of this assumption is proven (under the problem considered here) - connection of this approach to causal discovery is shown - the theorems lead to new estimation methods based on second-order statistics that allow ICA to be applied on Gaussian sources
Strengths
Strengths: - the paper is in general well written and of good quality - the assumptions on sparsity are more much more reasonable than in previous works and should allow for future works in this area. To this end theorems 2 and 3 provide relevant and useful comparison - theoretical assumptions are also often nicely illustrated with examples - some nice approach of transforming the theorems into algorithms in a justified manner (e.g. framing the search space of A; theorem 6) - connection between ica and causal discovery is well known and this paper further contributes to that similarity - addition of a necessary condition is a nice result and helps to understand the limits of this approach
Weaknesses
Weaknesses: The biggest conceptual problem I have is that the identifiability here is framed in terms of an optimization problem where the ground-truth covariance matrix is assumed. I find this idea hard to follow since identifiability should be a property of the data model. Typically the process is to assume that we have $\log p(x; \theta) = \log p(x; \hat{\theta})$ and then show that this implies $\theta \sim \hat{\theta}$ -- this approach makes sense as the starting point, equality of likelihoods, can be justified on the basis of MLE and its guarantees (at least in theory). Here instead one seems to start with the assumption that $AA' = \hat{A}\hat{A}'$. But the justification for this starting point is missing -- you should write it out fully. Also, in practice the empirical covariance matrix is used (eq. 8) but what are the guarantees here? I'm quite happy to revise my score upwards once this, and below, are discussed. I think the practical motivations for this paper are lacking and weak, perhaps partly due to a bit of unclear writing. For example, the authors write that "many biological traits ... are often normally distributed" to justify the importance of doing ica on Gaussian latents. But those listed biological signals are *not* typically latent, but often observed (after noise) and can usually be handled by standard ICA as there is no problem of Gaussian observations. Of course, it is possible to have latent biological signals too but that should be explained more carefully. It should be stressed however that the work still has important contribution in terms of fundamental research -- practical appeal does not always need to be obvious, so this is not necessarily a big problem. Authors also write that one possible situation in which their theorems are more useful than previous works, is when "an observed variable serves as a root cause". This may be possible but goes against the typical idea of ICA where observations are all assumed mixtures of independent latents. I feel this is however more easily understood in causal framwork. This leads to my next point... I find in general that ideologically this work is in some sense difficult to place as it feels like the ideas are more relevant to causal discovery yet it's framed as an ICA paper. But the assumptions on sparsity can still be quite restrictive and a substantially sparse mixing matrix, again, goes against the whole concept of ICA in some sense -- we are not really solving the mixing matrix problem if the mixing that happens is quite limited. Despite the examples given, it is still hard to understand exactly how restrictive sparsity assumptions are *in practice*. I find the estimation algorithm of the decomposition-based method in general a bit difficult to justify; optimally you would use an $l_o$ regularizer as this would agree with theory but that seems difficult to optimize. The likelihood-based method seems much more principled and I wonder if this paper could have been better framed around likelihood-based methodology in general. I'm surprised by the poor experimental results -- typically linear ICA achieves 0.99 correlation to the ground-truths eg. with FastICA so I would have expected similar results here. There are also no experiments on real data (not necessarily a big problem due to the breadth of theory here).
Questions
More concerns and things I'd expect revised for me to improve my score: line 16-17: the examples+reference here are pretty much straight form Hyvärinen (Independent Component Analysis: Algorithms and Applications, section 7). Perhaps with a bit more originality and effort some more varied, newer, examples or at least references could be provided? "distribution of Gaussian sources, the sparsity of the mixing matrix undergoes noticeable changes." Could you clarify what is meant here? Please include "links" to relevant proofs in the text. At the moment one has to wonder in the appendix to see whether the proofs exist. 329: "...based method (Eq. (8) or Eq. (9)) on data where both Assumptions 1 and 2 do not hold" The grammar could be polished as now it could mean "where assumption 1 and 2 do not hold at the same time" or that "neither assumptions 1 and 2 hold"
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Authors do indeed point out the limited applications to real world data. Authors also admit that " Since the true generating process of real-world data is inaccessible, it is challenging to quantitatively evaluate the applicability of these sparsity assumptions." however there is no discussion on why the experimental results are not as good as one would expect, this would be welcome.
Summary
This paper considers the problem of estimating a mixing matrix $A^*$, given measuremnts $x = A^* s$. The authors show that under Gaussian $s$, some assumptions on $A$, and using infinite $x$, the sparsest $A$ that satisfies $AA^{T} = A^* A^{* T}$ recovers $A^*$ upto permutation and sign of the columns. The authors also show connections to causal discovery in the case of structural equation models. Since the sparse norm is non-convex, the authors propose a convex relaxation using the $\ell_1$-norm that can be run in practice. The sparsity assumptions on $A$ induce polynomial constraints $H(A)$ on the elements of the matrix $\Sigma = AA^T$. These constraints are typically hard to compute, but the authors define an auxilliary function $h(A) = \text{tr} ( \sum_{k=2}^{n} off(A) \odot off(A) )$, where $off(A)$ is the off-diagonal version of $A$, and $\odot$ is the Hadamard product. The constaint set now becomes $h(A) = 0$, and the authors show that convex solvers can work in this setting. Minor: Using $H$ for the set of hard constraints and $h$ for the trace of the defined matrix is slightly confusing, please change it.
Strengths
- The authors show that their assumptions on the matrix are strictly weaker than existing approaches (Section 3.3) - The authors show connections to causal discovery, which I thought were interesting. - Existing approaches cannot handle Gaussian sources $s$, as this induces rotational symmetry on $AA^T$, and hence we can only recover $A$ upto rotation. In contrast, the current work can recover it upto permutation and sign of the matrix. - The authors are able to show that the convex relaxation in Section 4 recovers $A$ in the asymptotic limit of infinite samples of $x$. - Experimental results are nice.
Weaknesses
- The identifiability results are for infinite samples of $x$. - No runtime guarantees - The authors should spend more time commenting on the technical quality of the results. In some sense, the extension from the non-convex program to the convex relaxation is ``expected''. What new technical tools were required over traditional sparsity? - Experiments are a little simplistic, but that's fine for a theory paper.
Questions
- I'm not an expert on causal discovery. Are the connections between ICA and identifiability of structural equations established results? Relatedly, can you explain your contributions over existing work. - The authors should spend more time commenting on the technical quality of the results. In some sense, the extension from the non-convex program to the convex relaxation is ``expected''. What new technical tools were required over traditional sparsity? Without commenting on this, it would seem that choosing the right assumptions (1-3) are the key for this paper.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
N/A
Summary
In this paper, the authors develop new identifiability conditions for ICA, based on sparsity constraints. These assumptions, in particular, do not require the sources to be non-Gaussian, as is usually the case in ICA. Mostly, the authors take inspiration in previous work by Zheng et al, and weaken their main condition. Some assumption are shown to be necessary. As is usual with sparsity-based analyses, the resulting optimization problem is quite non-trivial, here the main difficulty is to explore the space of directed adjacency matrices whose corresponding graph is a DAG. The authors provide some discussion and implementation procedures about this. The proposed method is evaluated against FastICA on synthetic data. A (quick) ablation study is realized to examine the effect of the assumptions not being satisfied.
Strengths
- the contribution is interesting for the long-standing problem of non-Gaussian ICA. The authors make a commendable effort to discuss previous results, their main inspiration, and the strengthening of the results/weakening of assumptions - the paper is well-written and pedagogical, with examples and discussion scattered throughout about the assumptions and the results - some aspects of the paper are quite developed (relation with causal discovery, implementation, necessity of the assumptions...) without sacrificing the clarity of the writing despite the space constraints
Weaknesses
- the only assumption that less discussed than the other ones, in relation with previous work or its "realistic aspect" with respect to real data, seems to also be the "main" one, Assumption 2. The focus on hard constraints is theoretically understandable, but how realistic is it ? Also, the triple-presentation of the optimization problem is a bit redundant, the end problem being "just" optimizing among the space of DAGs - the paper focuses on exact identifiability, when the true covariance matrix is "available". This can be confusing for people not expert on ICA, can the effect of sample size and covariance estimation be discussed ? Does the resulting noise has an effect on "admissible" sparsity level (empirically, let's say), and for now the sparsity level $\| A\|_0$ does not seem to be constrained in any way (beyond assumption 1 and 2) - there is no discussion on computational complexity, it seems that the constraint $h(A)=0$ is very costly as it involves computing the $n$th power of an $n \times n$ matrix, at each iteration. Is there any simplification over brute force ? - no evaluation on real data. When the assumptions are only approximately satisfied, and there is strong noise on the covariance, it is not clear what is the performance of the proposed method
Questions
See "weaknesses" above Minor questions/typos: - l22 : "Kurtois" - l 95 : "statitsic" - Assumption 3 is a bit unclear at first read (it seems like a definition), only one sense of the "if and only if" is important, perhaps it is possible to reformulate - what is the exact sense of "asymptotically" in theorem 7 ? (see also question/comment about sample size, which is not discussed in the paper)
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors have discussed some aspects extensively, eg Assumption 1 with respect to previous work, other are a bit scarce, eg Assumption 2. Also, the computational complexity is not discussed, and there is no experiments on real data.
Response to rebuttal
I thank the authors for their careful rebuttal, which answers many of my concern. I will keep my score as is, but argue for accept.
Thanks
We are grateful to the reviewer for reading our response and for the acknowledgement. We will incorporate the suggestions in the revision. Many thanks.
I have re-read the paper and the other reviews, comments, rebuttals etc and have reconsidered my opinion. While my opinion of the paper is improved I am still also not fully convinced whether the paper quite merits acceptance. The theoretical work is very good and it does indeed relax conditions compared to previous work, but it I don't find it significant enough on its own as the conditions are still quite strong and restrictive and build quite clearly on previous ideas. I would thus expect more strong empirical results to really show me that "here we have a real-world problem and as you can see Zheng's approach fails, as do typical ICA approaches, but our model does much better". You do mention you are running some tests on Richard et al. experiments but I find those impossible to judge, as I have not seen them. **In conclusion: optimally, I would recommend the authors resubmit with more convincing empirical, real-world, results as I think it would make the paper a lot stronger but I will still engage in further discussion with other reviewers as to me this paper is very much on the threshold.** p.s. minor thing: **Q1** I think you missed my point here (and perhaps it was unclear on my part). I understand well where these are coming from e.g. as you say . *"In the large sample limit (as is the case for typical justification of MLE), one can show that $L(A)=L(\hat{A})$ implies $AA^T = \hat{A}^T \hat{A}$."* All I am saying you should probably say this more explicitly in the main text. This would make the connection to typical identifiability theorems explicit to readers.
We thank the reviewer for reading our response and for the further comment. We are glad that your opinion of the paper is improved, and you acknowledge that the theoretical work is very good. Regarding "build quite clearly on previous ideas", while the tool of sparsity is inspired by Zheng et al. (2022), we note that the technical development is entirely different. Our theory/proof leverages the distributional constraints of covariance matrices and effects of support rotations, thereby allowing us to relax the conditions. Such development and techniques were not seen in Zheng et al. (2022) (as well as the typical literature of ICA). Regarding "the conditions are still quite strong and restrictive", **this is precisely the motivation of our work**. We view it as our duty as researchers to progressively relax these conditions, thereby extending the applicability of the theory to cover more general mixing matrices. Due to the rotational invariance of the Gaussian distribution, ICA with Gaussian sources is a challenging ill-posed problem, and thus some assumptions are inevitably needed. As you acknowledged, we further showed that some of the assumptions are provably necessary, which shed light on the limit of these approaches. We admit that our work is primarily centered around theory. While we agree that the potential application of our theory in various real-world tasks would be exciting, we note that our theory can only be rigorously validated by ablation studies (via simulated data), which we have done in Section 5. This is because, as you also noticed, it is impossible to ensure that the unknown ground-truth data generating process satisfies some assumptions or not. Therefore, real-world experiments cannot play the role of validating the theory. Furthermore, it is worth noting that NeurIPS has traditionally published theory-focused papers if they offer relevant results and use nontrivial methods. We believe that both are the case. Regarding Q1, thanks for the clarification and we will make this more explicit in the main text, following your suggestion.
Just wanted to acknowledge that I've considered these comments but I hold my opinion of above that to me this is a borderline paper with important and good quality, theoretical advances but I also feel that they are not novel enough + suffer from strong restrictions (e.g. assumption 1). I don't buy the authors point that strong and restrictive conditions is only a good thing -- I think they show also that this approach is likely not practical. Yes simulated experiments are good at validating theory (and they could have been more extensive) I feel this is not strong enough of a 'theory paper' but would have needed experimental advances to show that it has practical value. Large number of previous, theoretical, NICA papers have managed to show the value of their algorithms on different types of real data. The more I think about the authors answers, the more I also worry about other issues such as how will these results hold when we want to estimate small number independent components for large data (observed dim >> latent dim) which is needed in a lot of situations.
Thanks a lot for the further response. Regarding "I don't buy the authors point that strong and restrictive conditions is only a good thing", it seems that there might be some misunderstandings w.r.t. to our previous response. We intended to say that, as you mentioned, since the conditions are quite strong and restrictive in this line of approaches (e.g. Zheng et al. (2022)), *“we view it as our duty as researchers to progressively **relax** these conditions”* to *“cover more general mixing matrices”*, which is the motivation of this work and has been emphasized throughout the paper. Specifically, some major conditions in this setting have been relaxed to necessary (weakest possible) ones. We did **not** say that strong and restrictive conditions are good things at all, and tried our best to make the existing ones as weak as possible. Regarding real data, we did not provide the final/full results in the original rebuttal due to the time constraint of the rebuttal period, and will report them in the revision. Thanks again. We are looking forward to knowing whether that has been clarified or not from your perspective.
I apologise for my misunderstanding. That makes sense thanks. I will keep my score as is since I don't really know how to take into account the work on real data. But I will make the case to the AC that if we can take such things (unseen work) into account then I would raise the score to borderline accept.
Thanks for taking into account the response and for discussing with the AC regarding updating the score.
Dear Reviewer ouDH, Thanks again for your time and efforts. In light of the comment regarding real data, we provide a summary of the preliminary results on real-world audio datasets. Specifically, we have tested ten independent speech audios (containing English, Japanese, etc), each of these has a sample size of $160,000$. The length of each audio is $10$ seconds. Using SparseICA (our method), we achieved an average MCC of $0.8788$ with a standard deviation of $0.12489$. Meanwhile, FastICA recorded an average MCC of $0.74211$ with a standard deviation of $0.09991$. Because according to the instructions by PC, we are not allowed to share any link in our response, **we have additionally sent the link of the result plot (violin plot) to the AC privately and let the AC decide whether the link can be shared**. Hope you can see it. Please let us know if you have any further questions. With best regards, Authors of #8190
Thank you -- Dear AC, would you be okay with sharing that figure. I believe it would be valuable as it would likely influence my final score of the paper. I think the authors have done great job trying to provide it in time so it would feel justified to share it.
A Kind Request for Further Feedback
Thanks again for taking the time to review our work. We have carefully considered your comments and provided responses to them. Since the discussion period will end in a few days, we would like to kindly request for further feedback. Could you please check whether the responses properly addressed your concern? Thank you very much.
Looking forward to your kind feedback
Dear Reviewer 6AWD, We are writing to kindly let you know that we have been eagerly waiting for your feedback on our rebuttal, despite your busy schedule. Since the discussion period will end on Monday, we hope for the opportunity to respond to your further comments or questions, if there are any. Any feedback would be appreciated. Thanks once again, Authors of #8190
A Kind Request for Further Feedback
Dear Reviewer 6AWD, We apologize for sending multiple reminders. Since the discussion period will end in two hours, we are very eager to get your feedback on our response. We understand that you are very busy. We would highly appreciate it if you could take into account our point-by-point response when updating the rating and having discussion with AC and other reviewers. Thanks for your time, Authors of #8190
Thank you for your responses and sorry for the late reply. I appreciate the authors kindly answered all of my questions. I choose to remain my score for now since there is no real data experiment result provided. I will raise my rating to 6 if you could provide a short summary (including figures and text summary) of the preliminary results on real data.
Dear Reviewer 6AWD, Many thanks for your further comment and considering our response. Here, we provide a summary of the preliminary results on real-world audio datasets. Specifically, we have tested ten independent speech audios (containing English, Japanese, etc), each of these has a sample size of $160,000$. The length of each audio is $10$ seconds. Using SparseICA (our method), we achieved an average MCC of $0.8788$ with a standard deviation of $0.12489$. Meanwhile, FastICA recorded an average MCC of $0.74211$ with a standard deviation of $0.09991$. Because according to the instructions by PC, we are not allowed to share any link in our response, **we have additionally sent the link of the result plot (violin plot) to the AC privately and let the AC decide whether the link can be shared**. Hope you can see it. Please let us know if you have any further questions. Hope for opportunities to respond to them. With best regards, Authors of #8190
Thank you for your fast response! It is promising to see SparseICA did perform well on real data. I have updated my score to 6. Please do include real data results in the revised manuscript.
Many thanks for taking into account the response and experiments. We will incorporate the suggestions and results in the revised manuscript.
Preliminary Results on Real-World Dataset
We are very grateful for all the insightful discussions. Due to the time limit and the specific requirement of Reviewer 6AWD and Reviewer ouHD, here we provide a summary of the preliminary results on real-world audio datasets. Specifically, we have tested ten independent speech audios (containing English, Japanese, etc), each of these has a sample size of $160,000$. The length of each audio is $10$ seconds. Using SparseICA (our method), we achieved an average MCC of $0.8788$ with a standard deviation of $0.12489$. Meanwhile, FastICA recorded an average MCC of $0.74211$ with a standard deviation of $0.09991$. Because according to the instructions by PC, we are not allowed to share any link in our response, **we have additionally sent the link of the result plot (violin plot) to the AC privately and let the AC decide whether the link can be shared**. Please kindly let us know if there are any further questions.
Decision
Accept (poster)