On the Consistency of Maximum Likelihood Estimation of Probabilistic Principal Component Analysis

Summary

This work studies the consistency of the maximum likelihood estimates of probabilistic PCA. These estimates are unique up to a rotation, so they use the quotient space and claim in their Lemma 6.1, ..., 6.5 stated in Redner[1981] are verified and apply the consistency results available in this book.

Strengths

This paper is overall well written, with a thorough introduction to equivalence classes and quotient topological spaces.

Weaknesses

Compared to Redner [1981] and Wald [1949], the novelty is not enough. I see the contribution as mainly an application of the result of Wald to the case of probabilistic PCA. The general idea is rather straightforward: using the quotient space instead of $R^d$ is the first idea that comes to mind. I agree that the precise formulation is not trivial but alone, and although the writing is excellent, it is not enough for a publication in my opinion.

Questions

Could you clarify the contributions of the paper ? Is there something I missed ?

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

1 poor

Limitations

Not applicable.

Summary

In this work, the authors propose a novel topological framework and show that the maximum likelihood (ML) solution of probabilistic principal component analysis (PPCA) is consistent in an appropriate quotient Euclidean space. The consistency results encompass more estimators beyond the ML solution. In addition, the ML solution has been shown to achieve strong consistency when the parameter space is compact.

Strengths

This work seems to be the first work to establish the (strong) consistency result of the ML solution of PPCA.

Weaknesses

It would be ideal to remove the assumption that the latent dimension $q$ is known.

Questions

I am not familiar with the techniques used in this work. Does the consistency in the appropriate quotient Euclidean space means that the ML estimate $(\hat{\mathbf{W}},\hat{\sigma}^2)$ converges to the true parameters $(\mathbf{W}_0, \sigma^2_0)$ in probability up to rotation (i.e., $\hat{\mathbf{W}}$ converges to $\mathbf{W}_0 \mathbf{R}$ with $\mathbf{R}$ being orthogonal)?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

NA

Summary

The paper discusses consistency of the maximum likelihood (ML) estimation in probabilistic principal component analysis (PPCA). Despite its wide applicability, proving ML estimation consistency in PPCA has been a challenging task because of the non-identifiability of the problem. The author(s) extend the quotient space topology idea presented in Redner (1981) to a more general setting where ML estimator is one such estimator. Also, strong consistency of the ML estimator is proved under a compactness assumption.

Strengths

1. Detailed description of the quotient space topology and the associated metric. The constructive approach was clear and concise. 2. With standard assumptions on the quotient space, the author(s) derived both weak and strong consistency of the ML estimator in the PPCA model. 3. Verification of the Wald's conditions [Wald, 1949] in the quotient space topology with sufficient technical details.

Weaknesses

1. Some remarks on the Wald's conditions on the quotient space to clarify how those conditions could be interpreted in the quotient space would be useful. 2. One or two examples to clarify the result in Theorem 4.2 would be helpful.

Questions

1. What would be $X_{\equiv}$ of Lemma 4.3 in the case of PPCA and in what sense $X_{\equiv}$ and the quotient $X\C$ are "identical" in that lemma? The two metric defined on those two topological spaces are not same- right? 2. I was wondering whether it's not possible to have the consistency result for the MLE in PPCA as a corollary of the results proved in the following Van der Vaart, A. W., & Wellner, J. A. (1992). Existence and consistency of maximum likelihood in upgraded mixture models. Journal of multivariate analysis, 43(1), 133-146. I understand the above may be for mixture models but can't we have a corollary for one mixing component which would be the marginal distribution of the data given W and $\sigma$ in your case.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Yes, the author(s) have adequately addressed the limitations.

Summary

paper addresses "Probabilistic PCA" which stands for a setup where the vector of p observations x can be written as x = Wz + eps where eps stands for an additive (Gaussian, centered, iid) noise, the matrix W \in R^{p x q} is an unknown and z in R^q is the other unknown . Only the value of q is known in advance and the goal is to recover W and z and remove the noise. The main challenge is to address the fact that the model is invariant w.r.t. applying a rotation R to W and z: Wz = (W R) (R' z) . Note that (W R) & (R' z) have the same euclidean norms as W & z. The contribution is to propose an analysis in the quotient space of the vector space by the rotations.

Strengths

The paper contains a rigorous consistency analysis of the PPCA problem with respect to rotation invariance

Weaknesses

The main focus of the paper on the rotation invariance does not seem to be the biggest concern with PPCA or other related (see Questions) methods.

Questions

1. There is a broader set of problems where rotation invariance causes trouble, and that set of problem directly related to the PPCA problem considered: all problems where the observations / measurements are of the form <u_i, u_j> or <u_i, v_j>. This comprises PPCA, but also matrix factorization and embedding problems. Can authors discuss how their method can expand to those problems? 2. Is there a relation between rotation invariance and computational complexity? Can a rotation be a distractor in the gradient steps convergence of a first order method?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

The focus seems to be relatively narrow and implications of the proven results not sufficiently emphasized.

Response to author rebuttal

I thank the author(s) for their detailed explanations in the rebuttal. I am happy with the answers they provided to my queries. I have increased my rating to "accept".

Correcting a minor typo

Hello everyone, It has come to our notice that there is a minor (and obvious) typo in the response we wrote for reviewer MhxU. In the second line of the rebuttal it should have been $\mathbb{P}(\widehat{W}\widehat{W}^T\to W_0W_0^T)=1$ and not $\mathbb{P}(\widehat{W}\widehat{W}^T\to \widehat{W}_0\widehat{W}_0^T)=1$.