>I have reviewed the rebuttal and acknowledge that accelerating linear system solving by NN within IPM iterations is a reasonable approach.
Thanks for your efforts and acknowledgement.
>While the use of NNs for solving linear systems is not new [1], integrating them within IPM solvers introduces specific challenges related to convergence and accuracy.
Thanks for sharing with us this work [1]. The authors of [1] focused on using NNs to solve **positive-definite linear systems of equations** while **linear systems in IPMs are indefinite**. We incorporate this work in our revised manuscript.
>I still have some concerns about the LSTM approach to solving linear systems with high accuracy.
>1. The provided theoretical guarantees for the Learning to Optimize (L2O) approach are limited to convex problems.
Thanks for raising this point. In this work, we focus on developing learning-based IPMs to address general nonlinear programs which might be non-convex.
- The **IPM itself (i.e., the outer loop of IPM-LSTM) would guarantee that the returned solutions are locally optimal** as long as linear systems are solved to specified tolerances, as shown by Proposition 1 in our manuscript.
- To solve linear systems via LSTM, we convert them to least squares problems of the form $\text{min}_y \frac{1}{2}|| J^ky+F^k ||^2$, which are **unconstrained convex programs**. As implied in Theorem 1 of [2], **solutions provided by** properly parameterized **LSTM networks would converge to one of the minimizers of convex optimization problems**.
Hence, from a theoretical point of view, **the proposed IPM-LSTM approach can solve both convex and non-convex programs to optimality**. We incorporate the above points in the revised manuscript.
>2. Many practical applications, especially those involving KKT systems for general non-convex problems, deal with ill-conditioned linear systems. Can the LSTM approach effectively handle such ill-conditioned systems.
Thank the reviewer for raising such an interesting point. Let $\kappa(\cdot)$ denote the condition number of a matrix.
- **The LSTM approach for solving linear systems is negatively affected by their large condition numbers.** To demonstrate this, we consider the least squares problem
\begin{align}
\underset{y \in \mathbb{R}^m}{\text{min}} \; f(y) := \frac{1}{2}|| J^ky+F^k ||^2.
\end{align}
We utilize a first-order method (say the steepest descent method) to minimize $f(y)$ and achieve a linear convergence rate [3], i.e.,
\begin{align}
f\left(x_{k+1}\right) - f\left(x^{\star}\right) \leq \left(1-\frac{2}{(\kappa(J^k))^2+1}\right)^{2} \left( f(x_{k})-f(x^{\star})\right).
\end{align}
As we discussed in the **Preconditioning** part of our manuscript, since solving linear systems via LSTM networks emulates iterative first-order methods, thus the value of $\kappa(J^k)$ affects the performance of LSTM networks.
- As shown in the computational studies (Section 3.1) of [4], **LSTM networks can empirically achieve a faster convergence rate than traditional first-order algorithms** when solving the same least squares problems.
- To alleviate the effect of large condition numbers, as discussed in the **Preconditioning** part of our manuscript, **we have employed preconditioning techniques**.
>3. It is also unclear whether the used simple non-convex problem is ill-conditioned or not.
Thanks for your suggestion. For the simple non-convex programs used in our experiment, we report $\kappa(J^k)$ and their values after preconditioning (in parantheses) across several IPM iterations (say 1, 10, 20, 50, 100) in the following table.
- The condition numbers **$\kappa(J^k)$ remain within reasonable magnitudes**, even during the later IPM iterations.
- Applying the **preconditioning** technique indeed significantly **reduces the condition numbers** for those non-convex problems.
| Instance | $1^{\text{st}}$ Iter. | $10^{\text{th}}$ | $20^{\text{th}}$ | $50^{\text{th}}$ | $100^{\text{th}}$ Iter. |
| ------------------------- | ------- | --------- | --------- | --------- | ---------- |
|**Non-convex Programs (RHS) (100, 50, 50)**|53.8(59.8)|126.2(580.7)|153.1(711.2)|208.7(1004.8)|348.4(1860.1)|
|**Non-convex Programs (ALL) (100, 50, 50)**|55.4(59.8)|113.6(517.0)|139.8(658.5)|214.0(1190.9)|329.2(1859.9)|
|**Non-convex Programs (RHS) (200, 100, 100)**|91.5(99.8)|157.1(1114.0)|205.8(1441.1)|326.2(2398.3)|488.3(3667.8)|
|**Non-convex Programs (ALL) (200, 100, 100)**|72.1(75.7)|175.4(1143.4)|184.5(1352.7)|249.5(2016.6)|368.4(3015.3)
[2] Liu, Jialin, et al. (2023) "Towards constituting mathematical structures for learning to optimize." ICML.
[3] Nocedal, J., & Wright, S. J. (Eds.). (1999). Numerical optimization. New York, NY: Springer New York.
[4] Andrychowicz, M., Denil, M., Gomez, S., Hoffman, M. W., Pfau, D., Schaul, T., ... & De Freitas, N. (2016). Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29.