Summary
This paper explores the performance of an optimal (in the Bayes-Optimal sense) estimator in the teacher-student setting with aligned architectures being one-hidden layer neural networks with quadratic activations. The number of neurons is extensive in the dimension. Given previous works where the "proportional limit" regime was studied (i.e. asymptotically square dataset), finding that a vanilla estimator could attain optimal performance, they focus on a slightly different regime. Precisely, the author(s) build upon numerical observations of other works and let the number of samples scale quadratically in the dimension.
Having chosen the playground, the analysis proceeds with recasting the problem to a form that exploits the symmetries of the objective function, and a replica computation of the free energy. Such computations allow the author(s) to derive an asymptotic formula of the Minimum Mean Square Error.
To back the Physics-based derivation, they discuss a break-down of the steps required to formalize it completely. Of the three main points, two are conjectures. The final one instead has a mathematical proof.
Given that Gradient Descent is the go-to method, they explore its performance in relation to the theoretically optimal MMSE. Interestingly, they find that in the noiseless case it appears to sample weights from the posterior. The same picture breaks when the labels have noise. There, they discuss the appearance of a phase transition that trivializes the randomness over initialization of GD.
Strengths
- Equation (1) and its natural link to the degrees of freedom of the target function is an appealing result.
- The connections with ellipsoid fitting, computations in random matrix theory, formalized results on neural networks, and extensive-rank matrix denoising validate it. In simple words: it is not a made-up paper, but it is well positioned in between earlier works.
- Understanding industry standard is crucial. Gradient Descent is the idealized version of what is actually used in practice. It is therefore valuable to understand it, as the author(s) did.
- The result about GD opens up to continuations of the work into such direction of understanding formally what is happening there. This paper is not a dead-end.
- Section 4 is an interesting take on trying to vindicate the replica method. I believe there is value in trying to break down what would be needed to make the Physics derivation a mathematically rigorous one. At the same time, it is of independent interest to use the replica method, as it tells us what we have to prove. Anyways, nice idea.
- I was kind of surprised to see that as $\kappa$ increases the MMSE worsens.
- The techniques used are, in my honest opinion, beautiful and effective.
- Even if the MMSE cannot be simulated (unless an algorithm achieving it was known), the author(s) bring to the table experiments for GD that are to the point and motivate further work.
- The references are, to my limited single human reading stack, to the point.
- The weaknesses I will discuss below are to be understood in the sense that I understand that this is a paper with lots of Physics-flavour. Personally, I agree with the modelling idea of finding a solvable model, hence the choice of a "simplified setting". Also, this simplified setting already requires non-trivial computations, and is therefore worth exploring. Nevertheless, I will outline what worries me to stimulate the discussion.
Overall, I believe this was a good read. Thanks to the author(s)!
Weaknesses
As mentioned in lines 350-353, the setting is very restricted. While universality might come in help, at least for conjecturing that some aspects extend nicely in the high dimensional limit, there are at least three features that in my opinion are more complicated. Below, I will elaborate on them.
1. Perhaps the solvable one. You consider matching architectures, with $m = m^{\star}$, I wonder, what is the technical challenge in analyzing the case in which the activations are the same but $m\neq m^{\star}$?
2. Your target function is "neuron agnostic" in the sense that it takes a simple mean of the teacher neurons. How would your technique behave when we take a one hidden-neuron that also has random Gaussian weights $\{a_k\}_{k = 1}^m$ in the second layer? I have not thought this through extensively.
3. The most challenging: a quadratic activation. While the Statistical Physics literature focuses a lot on quadratic activations/phase retrieval, which is thought to be an interesting problem, I am concerned about the extension in this setting to other activations. It appears to me that when recasting the task to a matrix estimation problem (lines 175-180), one heavily relies on "expanding the square" (lines 175). In other words, the symmetry of the activation function naturally builds the matrix $\mathbf{S}^{\star}$ (lines 176-177). Considering other (e.g. non-polynomial, or even non-square) activations, it looks to me as a very hard task. I do appreciate that the author(s) recognized this, but would be even happier if, in the spirit of section 4, the bonus page for publication were spent to discuss how a quadratic activation is not a dead-end.
More in depth, can you bring over an argument that can convince the reader that the method is worth pushing over the boundary of something that allows you to over-use Gaussian integrals (in the sense that square activations are perhaps the easiest non-linearity)?
The second concern is mostly structural and philosophical. I am convinced that all the results of the paper are of interest to the community. However, from my first reads, I had the impression that the "Replica argument", its "formalization" and the "analysis of GD" had little meaning if placed all inside a conference paper.
For example, while Section 4 is very useful for grounding the replica method, I only appreciate its value in the sense that it connects the dots with other works. In some sense, I find this counter-intuition that there are very few resources on Gaussian Equivalence and modern Statistical Physics of neural networks that discuss this, and then find a nice Section clearing out some aspects in a paper that should be 9 pages. I do wish at some point that these scattered nice sections will be condensated in some resource that will be accessible (e.g. a book).
Similarly, the analysis of GD could be justified for its industrial importance, and even simply because it is the quickest algorithm to compare with the MMSE prediction, but at the same time it poses the question: "What is the soul of this work?".
Please note that this weakness is more provocative than actually requiring action.
Questions
Can you please make the legend in Figure 2 (right) more explicative? I obviously get it, but do not understand why you chose to "condensate it". As a reader, the others are easier to parse, as they label everything.
- The regularization of Figure 3 is never discussed in text, why do you add it? Just to show that the distance from the MMSE becomes larger? What exactly counts to you as "trivialize the gradient descent landscape almost completely" in Figure 3 (Left), as you claim in lines 802-803? Please note that I understand the value of Figure 3 (right).
- Related to the point above, why do you seek to show regularized objectives in the first place? I mean, what motivates your experiment?
- Please, see the first weakness, where I planted some questions.
### Typos
I will write down in this subsection what I noticed by reading. Take it as a friendly contribution. All of them are largely not relevant to the evaluation of the paper.
- The instruction block for the checklist was not deleted.
- (line 161) you write "student", but I noticed that in other instances you used a different way of placing quotation marks, that is the TeX standard.
- Equation (5) lacks punctuation.
- (line 222) the end of the sentence "using probabilistic technical amenable to rigorous treatment" probably changed wording, maybe "technical"was meant to be "techniques"?
- (line 779) the "root" quotation marks.
- (line 792) is $30,\,000$ really the right number? Why three zeros? Maybe I am just not used to this standard.