Fast Projected Newton-like Method for Precision Matrix Estimation under Total Positivity

We study the problem of estimating precision matrices in Gaussian distributions that are multivariate totally positive of order two ($\mathrm{MTP}_2$). The precision matrix in such a distribution is an M-matrix. This problem can be formulated as a sign-constrained log-determinant program. Current algorithms are designed using the block coordinate descent method or the proximal point algorithm, which becomes computationally challenging in high-dimensional cases due to the requirement to solve numerous nonnegative quadratic programs or large-scale linear systems. To address this issue, we propose a novel algorithm based on the two-metric projection method, incorporating a carefully designed search direction and variable partitioning scheme. Our algorithm substantially reduces computational complexity, and its theoretical convergence is established. Experimental results on synthetic and real-world datasets demonstrate that our proposed algorithm provides a significant improvement in computational efficiency compared to the state-of-the-art methods.

Paper

Similar papers

Peer review

Reviewer oo936/10 · confidence 3/52023-06-20

Summary

This paper proposed a novel algorithm for solving the problem of estimating precision matrices in Gaussian distributions with MTP2 constraint. The problem formulation the paper considered is a sign-constrained log-determinant program with disconnectivity constraints. The algorithm is based on a carefully designed search direction and variable partitioning scheme. The author provide the choices of approximate Newton direction and the computing step size with the theoretical convergence analysis, where the author finally shows the convergence in finite iterations. The experimental results indicates better computational efficiency of the proposed algorithm compared to the state-of-the-art methods.

Strengths

The paper is well written and clear, the approach is well supported by the theoretical analysis. The experiment and application seem strong and nice. The authors also made comparisons to other state-of-the-art methods that makes the paper more complete.

Weaknesses

- Some quantities could be elaborated in the main paper. e.g., $D_k$ in line 153. I am not sure if the form of the diagonal matrix is important or a minor issue that will only be useful for theoretical analysis. - It might also be good to mention the special case of the algorithm without the disconnectivity constraint and how it compares with other existing methods. - It seems to me that the choice of regularization parameter $\lambda_{ij}$ doesn't affect the results in Theorem 4.4 & 4.5. Could you elaborate more on this?

Questions

See weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have adequately addressed the limitations.

Reviewer KQq47/10 · confidence 3/52023-07-03

Summary

This paper presents a novel projected Newton-like method for computing the inverse covariance matrix of a multivariate Gaussian distribution, for which the off-diagonal elements are non-positive. The method is shown to be an order of magnitude faster than existing state-of-the-art methods for this task, while the authors also prove convergence to the global optimum.

Strengths

Well-written. Fast algorithm. Complete theory. Convincing experiments.

Weaknesses

I would like the authors to spend a little more time motivating and introducing the problem. In particular, discussing the following questions on the level of introduction, might make the paper more appealing. 1. What features of the data lead to such an inverse covariance matrix structure? 2. What is the exact "reduction on the number of observations" for a model with the MTP_2 structure? (line 23). 3. What is the role, in terms of data / applications, of the disconnectivity set Epsilon? 4. Why is problem (1) preferred to other alternatives?

Questions

Please see above.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

-

Reviewer 7MEo3/10 · confidence 4/52023-07-06

Summary

(1) This paper investigates the problem of estimating precision matrices in Gaussian distributions  that are multivariate totally positive of order two (MTP2), and propose an algorithm based on the two-metric projection method. (2) Empirical tests are conducted.

Strengths

- **Originality:** The techniques used in this paper reduces computational complexity and its theoretical convergence is established.   - **Quality:** The notations, problem statements, and mathematical proofs appear rigorous, although I have not scrutinized them word by word.   - **Clarity:** The readability of this article is good.   - **Significance:** The techniques (and tricks) may inspire and guide future researchers.

Weaknesses

1. The main issue with this paper is that it does not cite the highly relevant paper [A]. The bound provided by theoretical analysis of this paper is a supremum of a limit point, which generally can not guarantee the convergence, while the theoretical result of [A] is much more attractive. 2. Also, in the empirical study, the algorithm presented in the paper has not been compared to [A].

 [A] Ying, Jiaxi, José Vinícius De Miranda Cardoso, and Daniel P. Palomar. "Adaptive Estimation of Graphical Models under Total Positivity." (2023).

Questions

Please address these questions I posed to in the previous message.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

Yes, they have.

Reviewer AisR7/10 · confidence 4/52023-07-07

Summary

The authors propose and analyze a variable metric projected gradient method for a variant of the graphical Lasso problem where the off-diagonal elements are additionally constraint to be non-positive, and in which specific entries of the precision matrix are known to be zero. The proposed metric which is used to precondition the gradients comes from the curvature of the function $X \mapsto -\operatorname{log det}(X)$ , which is part of the objective, and thus incorporates second-order information of the objective function, unlike more standard approaches for the problem. The authors put their result in the context of the recent literature, and show in numerical experiments on covariances created from multivariate Gaussian whose precision matrix sparsity comes from a Barabási-Albert random graph that the proposed method outperforms other methods used to optimize the same objective function, such as block coordinate descent or the proximal point algorithm in terms of clock-time convergence speed. From a theoretical point of view, the authors show that the proposed algorithm indeed converges to the unique minimizer of the (convex) objective, that the proposed stepsize rule indeed works, and that the algorithm identifies the true support in a finite number of iterations under an additional reasonable assumption on the minimizer.

Strengths

I consider the proposed efficient estimator for the sparse precision matrix with off-diagonal non-negativity constraint of interest for the machine learning community due to wide potential applicability of the setting to engineering problems. The specific gradient preconditioning through the weighted metric induced by the second derivatives of the $\ell_2$-error is original for the problem, and the computational advantages compared to state-of-the-art methods are well explained ($O(p^3)$ time complexity) and showcased in sufficiently timing experiments. The choice of the comparison methods is adequate and the experimental setups are informative. The paper is clearly and well written. A strength of the paper is the suitable co-design that heavily takes into account the problem structure, which makes the Hessian approximation used in the variable metric preconditioning simple enough to compute in a reasonable amount of time, while at the same time delivering a benefit in per-iteration improvement.

Weaknesses

While the problem as formulated in (1) is convex as stated, the underlying estimator pipeline is somewhat inherently non-convex due to the fact that the $\lambda_{ij}$ correspond to weights of a reweighted $\ell_1$-term whose weights are chosen based on the MLE. This detail could be mentioned more prominently in the paper. From a computational point of view, the line search method deployed has the computational disadvantage that a lot of function evaluations, which might be expensive due to their need for spectral decompositions, might be needed to make the method work. It remains unclear whether the convergence rate result Theorem 4.5 is optimal. Due to the incorporation of second-order information, a discussion of why superlinear convergence cannot be shown or observed would be further informative.

Questions

- When discussing the positive definiteness constraint in line 112, you mention that it is not closed and furthermore, cannot be managed by a projection. However, by looking at (1), it seems that due to the objective, it is clear that the minimizer will not be singular due to the existence of the term $- \operatorname{log det}(X)$ in the minimization: If at least one of the eigenvalues of X is 0, $\operatorname{log det}(X)$ is finite. Lemma B.1 touches upon this, but couldn't the positive definiteness constraint be relaxed to a psd constraint? - In the considered model (1), the role of the disconnectivity constraint remains slightly unclear. Appendix A.1. and also the discussion of the financial time series example leaves the impression that the disconnectivity set is found by a similar method as the weights of the $\ell_1$-term $\lambda_{ij}$. Can you make the distinction clearer? - In Theorem 4.2, it will be convenient if one the authors can refer definition equation for $L_f$.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The this discussion of limitations is adequate in general. However, it would have been beneficial if the "hidden - $\lambda$-dependence" on the sparsity of the estimated covariance matrix had been explained better in the paper.

Reviewer KQq42023-08-12

Thank you for the detailed responses.

Reviewer AisR2023-08-16

I thank the authors for their response, which clarifies in particular the computational aspects involving the line search and Cholesky decomposition. We recommend that the indicated remarks will indeed be included into the final manuscript. Following up on the discussion of the PSD constraint: It seems then to me that working with the PSD set instead of the PD set would not change anything in the algorithm if we kept the line search, as the positive definiteness is automatically enforced due to the form of the objective $f(\mathbf{X})$. If that is the case, we recommend mentioning that. Regarding the point about superlinear convergence: Could you elaborate on what the algorithm would look like if the exact Hessian was taken, and quantify how this makes the computational burden considerably worse? Maybe I am missing something, but the formula of Proposition 3.1. seems to be computationally of the same order as the formula of Proposition 3.3. Thank you!

Authorsrebuttal2023-08-16

We sincerely appreciate your thorough review and insightful comments. You are correct that using the PSD set, rather than the PD set, would not alter the algorithm if we continue to use the line search method to handle it. We recognize the importance of this clarification and will ensure to include this point in the final manuscript. In response to your query regarding the use of the exact Hessian, if we were to employ the exact Hessian, specifically the Newton direction from Proposition 3.1, the algorithm's convergence to the minimizer cannot be guaranteed, despite its computational efficiency. This is primarily due to the fact that such a direction may not always serve as a descent direction. As depicted in Figure 1(c) of our paper, the Newton direction from Proposition 3.1 stops decreasing the objective function value after a few iterations, thereby failing to converge to the minimizer. If our goal is superlinear convergence, we would need to replace $[H _k^{-1}] _{\mathcal{I} _k^c\mathcal{I} _k^c}$ with $\left([H _k] _{\mathcal{I} _k^c\mathcal{I} _k^c}\right)^{-1}$ in equation (11). However, it is important to note that this modification would significantly increase the computational burden compared to our method, as it cannot leverage the special structure of the Hessian to lessen the computational load. Moreover, the size of $\mathcal{I}_k^c$ could become large during iterations, which is not within our control. This could lead to a per-iteration complexity of up to $O(p^6)$. In contrast, our method guarantees a complexity of $O(p^3)$ per iteration. We hope these explanations sufficiently address your inquiries. We once again express our sincere gratitude for your valuable feedback. To further enhance the clarity of our work, we plan to incorporate all these discussions related to line search, Cholesky decomposition, the exact Hessian, and superlinear convergence into the revised manuscript.

Reviewer oo932023-08-16

I would like to thank the authors for their reply. My assessment remains inclined to the positive.

Authorsrebuttal2023-08-18

We sincerely appreciate the time and effort you've dedicated to reviewing our paper and considering our responses. Please be assured that we have concentrated our efforts to thoroughly address your concerns about paper [A], through both the individual response above and the global response. If you find our responses to be satisfactory, it would be our honor if you could consider increasing the review score. We warmly welcome any further questions or concerns that you may have.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC