Summary
This work proposes a modification of the original CHFICA characteristic function to the noisy ICA case without requiring knowledge of the noise distribution parameters.
The modified independence score is then used to select the best out of multiple ICA methods based on the score assigned to their solutions, in a per dataset basis. In practice, an average score is evaluated over random directions t, which renders the score computationally inefficient for direct Minimization.
The empirical estimate of the new independence score is shown to converge to the population score if the data covariance is concentrated.
Empirical evidence is shown that the new empirical score correlates well with the well-established Amari error.
This work also introduces two computationally efficient contrast functions to be Maximized (CHF-based and CGF-based) for BSS, both of which do not require higher moments and, thus, work well on data with near-zero kurtosis, like Bernoulli(p) with low p.
Theoretical analyses of local and global convergence are also included for a large class of smooth contrast functions that meet certain desirable properties (Assumption 1). Properties (a) and (c) are critical as they yield contrast functions that are not influenced by additive independent Gaussian noise.
The CHF contrast only requires a finite second moment and is suitable for heavy-tailed sources.
The CGF contrast is not appropriate for heavy-tailed sources.
Global convergence: The Hessian of both contrasts is shown to have the form C = BDB^T. The maxima of the Hessian-adjusted contrasts is shown to occur at B^-1 u = e_i for positive elements of D. Fast sequential power method optimization in a pseudo-Euclidean space defined by C^-1 was shown by [48]. Before optimization, C need only be computed once for some random vector u and just reused during the power iteration updates. A requirement of no sign change of the third derivative of the contrast over the half-lines is required to sufficiently distinguish between Gaussian and near-0 kurtosis source distributions.
Local convergence: Linear convergence is attainable without the third derivative requirement. Geometric convergence is attained at some low epsilon based on how non-gaussian the source is.
Contrary to prior work on cumulant-based methods, convergence of contrasts meeting Assumption 1 was established based on the characterization of the contrasts in the pseudo-Euclidean space and not on the convergence of the power method.
Experimental results support Meta's ability to select the optimal solution over various algorithms, showing variance reduction in the estimates, uniform performance across high and near-0 kurtosis.
Evaluation of the CHF and CGF contrasts show that CHF outperforms other methods at higher noise, but only at sample size n > 20000.
The Meta algorithm picks up the slack of CHF by selecting the competing PFICA solution when n < 20000.
Strengths
Originality :
- The work introduces a somewhat novel combination of known ideas in a reasonably novel formulation that adapts previous work to the case of additive Gaussian noise.
- The work differs from and extends previous contributions, dealing with near-zero kurtosis sources and additive Gaussian noise.
- The manuscript cites related work on contrast functions for BSS and noiseless ICA methods, adequately indicating the sources of inspiration.
Quality:
- The work is technically sound, with thorough proofs.
- The theoretical claims are well supported by the experiments.
- Strengths and weaknesses are discussed well.
Clarity:
- The work is written well overall, focusing on the key points and contributions.
Significance :
- The results are important as they establish a consistent framework to select the optimal solution over a range of BSS methods for noisy ICA based on the proposed modified independence score.
Weaknesses
Quality:
- The code is not friendly to readers as there is no 1-to-1 correspondence with the notation in the paper. Suggestion: improve documentation and tidy up the codebase (especially comments and function arguments) for readability.
Clarity:
- Despite the well-written manuscript and thorough methodology and proofs, the paper contains many typos (both in text and proofs) and inconsistencies in notation (especially between Appendix and main manuscript) that limit the clarity. Besides fixing these typos, explicitly include some of the "obvious" steps currently omitted in the derivation; that would greatly improve readability. Also add some high-level intuitions as noted below.
- Clarify the origin of new contrast functions and how they came about. It is unclear what motivated or led to these otherwise "semi-arbitrary" functions.
- Define O_P(.)
- It appears that the new independence score does not meet some of the properties in Assumption 1. Please, clarify if that is indeed the case and, if so, which properties specifically.
- Appendix A.3: Clarify that Algo 2 is NOT specific to contrasts based on power iteration only, but any sequential method. It is suggested to make one separate sub-section covering how the projection is done and another for the Algo description itself. Also clarify what the "Total" (non-sequential) Algo would look like, for comparison.
- Lines 119-122: t.T should be bold face.
- Eq. 2: missing last t in the PRODUCT term.
- Section 3.2: Change "Properties of Contrast Functions 1." to "Assumption 1. (Properties of Contrast Functions)"
- Line 175: remove "them"
- Line 182: "note" --> "not"
- Footnote 2 (page 5): add "and remains a linear combination of z"
- Theorem 4, line 254: Assumption 1(d) is NOT about the third derivative.
- Line 271: Define the value of k in this experiment. I see 11 mentioned in passing in line 786...
- Line 277: Fix contradicting statements: "CHF and CGF are initialized using the B estimated via PFICA" and "CHF and CGF is based on a single random initialization".
- Line 283: "also used" --> "also be used"
- Table 1 caption: Clarify that since median is being reported, Meta is better than the best method “on average”, but on any individual experiment it is identical to the best method.
- Line 299: refs to 2b and 2c are in flipped order.
- Line 315: In the noise power experiment, it is stated that "the difference between the two leading algorithms is small." Therefore, the same acknowledgement should be indicated in line 315 after "CHF dominates, and one can see that Meta starts following CHF"
- Line 456: Assumption 1(d) is NOT about the 3rd derivative constraint...
- Line 464: Missing ' in g.
- Line 472: For consistent notation, it should be Delta(t,F | P) = 0, not Delta_F(t) = 0
- Lines 474-477: there is an extra transpose symbol at the last t (several times).
- Line 510: (e) --> (d)
- Notation and conditions in lines 524-529 is not consistent with Theorem 3 in the main manuscript.
- Line 777: w_i is not defined, g_i should be ~g_i.
- Lines 784-789: the r defined here has no relation to the r in Algo 2. Please pick another letter.
- Algo 2:
- Clarify why it is necessary to evaluate Y_j per component and why it would not be equivalent to just compute W = X V^T U^T r
- for j in range[1, M] : M is not defined, but I think it would be l + 1? Also, the letter j is used for many different things inside the same loop. Change to " for p in range[1, l + 1] ", and s(j) --> s(p).
- " W(:,j) ← Y_j r " --> " W(:,a) ← Y_a r "
- I suspect r is equivalent to t in Eq. 2, if so change it to t.
- Appendix A.4: Verify that all occurrences of Assumption 1 (d) are correct (i.e., referring to symmetry not the third derivative)
Questions
Questions were included along with the weaknesses.
Limitations
Yes, the authors addressed the limitations of the work.