Algorithmic Regularization in Over-parameterized Matrix Sensing and Neural Networks with Quadratic Activations

We study the problem of recovering a low-rank matrix $X^\star$ from linear measurements using an over-parameterized model. We parameterize the rank-$r$ matrix $X^\star$ by $UU^\top$ where $U\in \mathbb{R}^{d\times d}$ is a square matrix, whereas the number of linear measurements is much less than $d^2$. We show that with $\tilde{O}(dr^{2})$ random linear measurements, the gradient descent on the squared loss, starting from a small initialization, recovers $X^\star$ approximately in $\tilde{O}(\sqrt{r})$ iterations. The results solve the conjecture of Gunasekar et al. under the restricted isometry property, and demonstrate that the training algorithm can provide an implicit regularization for non-linear matrix factorization models.