Response 2
> In $\mu$P for gradient descent, the learning rate is set to $\eta_0 \gamma^2$, where $\gamma=\mathcal{O}(N^{0.5})$. This parametrization of the network is such that the magnitude of feature updates is order 1 for all layers.
Depends on the implementation; when not using any multipliers (Table 3 implementation variant in [1]) the learning rates scale as $\Theta(N)$, $\Theta(1)$, or $\Theta(1/N)$ for input, hidden and output layers respectively.
> What does the reviewer mean that different weights move at different rates in ? Perhaps a simple example would help us understand.
Generally, since the changes input to any layer (activations/features) in $\mu$P will be order $\Theta(1)$, and there are $\Theta(N)$, that means that the change to the weights $\Delta W$ (resulting from a gradient step with a muP learning rate) must be roughly $\Theta(1/\sqrt{N})$ to ensure the output pre-activations are also $\Theta(1)$. However, this is not the case for the input layer; there are only $\Theta(1)$ inputs (rather than $\Theta(N)$) for the input layer. Hence, the size of the change $\Delta W$ to the input weights should also be order $\Theta(1)$ to ensure the entries of the output of that layer are $\Theta(1)$. That means there should be a discrepancy in the size of the changes $\Delta W$ to the weight matrices in the hidden and last layer ($\Theta(1/\sqrt{N})$) and input matrices $\Theta(1)$.
Similar reasoning applies to biases, where the change after a gradient update $\Delta b$ will generally be $\Theta(1)$ as well.
Of course, the above applies to changes to weights and biases (i.e. gradients multiplied by per-layer learning rates), and not to the gradients (which haven't been multiplied by a learning rate yet). In general, these will also have different scales for different layers (input, hidden, last).
Now, this becomes doubly weird when considering something like the gradient vector or Hessian matrix for the entire network. There, you will have elements that grow at different rates with width (depending on whether they are weights or biases for an input, hidden or last layer), and also the *proportion* the elements will be different for different layers. For example, the number of the elements in the gradient vector corresponding to weights of a hidden layer grows as $N^2$, but for the input and last layers the number of elements grows as $N^1$. Hence, for large enough widths almost all elements of a gradient vector or Hessian matrix will correspond to hidden layer weights.
---
> A clarification: what does the Reviewer mean by eigenvalues of a gradient here (which is a vector)?
I mean the gradient $\frac{\partial L}{\partial W^{(\ell)}}$ for a particular weight matrix $W^{(\ell)}$, reshaped to have the same shape as that weight matrix. If it's not square, I mean the singular values.
To be more specific, I did actually have the weight update $\Delta W^{(\ell)}$ in mind (i.e. the gradient multiplied by the learning rate in the case of SGD), since I think in that case, the $\mathcal{\Theta}(1/\sqrt{N})$ scaling for hidden layer weights means that the spectrum should converge to something well-defined, and for others it might not (although I'm not 100\% sure about this claim, I haven't worked through the maths in detail).
### muP check
As an aside, did the authors do a muP coordinate check, as recommended in Appendix D.1 and Figure 5 of [1]? Speaking from personal experience, it's quite easy to mess-up the implementation of muP as there are many small caveats to the general rules (e.g. how to initialise LayerNorm parameters), and sometimes one can get learning rate transfer even when there are minor mistakes in the implementation. The coordinate checks usually reveal those. I think adding these to an appendix would make the authors' empirical claims significantly more resistant to any potential readers' doubts.
[1] Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer