Rebuttal
We thank the referee for their careful consideration and feedback.
**Weakness 1**
We agree with the reviewer that architectural improvements are likely a big contributor to the phenomenon we observe. In fact, in Appendix D, we list architectural characteristics as a possible factor to the behavior we observe, and discuss it (see architectural characteristics paragraph). While we observe that the stable behavior of cosine similarity is, in fact, preserved on simple architectures (vanilla MLP with ReLU activations), our experiments cannot conclude on the contribution of e.g. batch norm, skip connections, etc.
We agree this is an interesting line of research for future work. However, as our experiments are quite expensive (requires double computation as each training run needs to be computed twice), the range of ablations we have conducted is already quite extensive.
Most importantly, the main goal of our work is to characterize the properties of neural loss landscapes in practical settings, i.e. using modern architectures. As we discuss, our main goal is not to explain why these properties are present, but to quantify them and their relation with various factors (such as optimizers, depth and width, seeds, task, etc...). Since these properties seem very relevant to training dynamics, better understanding them mean we can better understand training behavior, and build a more practically relevant theory. Thus, while indeed our observations are only validated on heavily engineered architectures, we argue that this instead makes our work more relevant, and validating our claims on simple architectures that are not deployed in practice would be less meaningful.
**Weakness 2**
The values of RSI and EB appear close to $0$ in Figure 2 due to being small compared to the late stages of training, where both RSI and EB go to infinity (see late training behavior paragraph). But this is just a matter of scale, and the cosine similarity in fact quickly stabilizes. The speed at which the iterates $w_t$ approach $w_T$ is in fact entirely determined by RSI, EB, and the learning rate (see Equation 4), and the value of RSI and EB in the early stages, while small in comparaison to late training, are not negligible. For a fixed cosine similarity $\frac{RSI}{EB}$, observing smaller values of RSI and EB simply means the ideal learning rate should be larger.
"RSI and EB follow predictable trends" does mean that the optimization trajectory advances toward its last convergence point at a steady pace, and that despite non-convexity and stochasticity, all iterations (with very few exceptions) contribute similarly in term of parameter space progression. The stability of the values of RSI and EB across minibatches, and its predictability across phases of training, mean that ideal values of the LR are stable across minibatches, and predictable across phases of training. This explains why simple LR schedules can be so efficient in training neural networks despite their non-convexity and the stochasticity of gradient sampling.
**Weakness 3**
We do compare to the baseline where gradients are replaced by noise, as suggested by the referee, in Section 6 (paragraph "induced Bias"). We also discuss the low value of cosine similarity in Section 4 (paragraph "low value of cosine similarity") and remark that in fact, if it wasn't very low, we would be able to train neural networks in very few steps, which know not to be the case. The cosine similarity being low can be compared to a quadratic function being badly conditioned: optimization is difficult, but follows a simple underlying structure, which eventually drives the convergence. Interestingly, RSI and EB would also follow predictable trends in the case of a random walk due to the high dimensionality having a regularizing effect. This can be interpreted as random walks having a stationary update rule. The complex structure of neural networks suggest the training dynamics could be highly complex (see contrasting examples in Section 6). Our work finds that it is not the case.
**Weakness 4**
As stated in our reply to Weakness 1, our work does not provide direct insights on which architectural features enable such properties. The goal of our work is to improve our understanding of the fundamental properties of neural loss landscapes that make first order optimization so surprisingly efficient, rather than exploring their origin. Once a sufficiently refined characterization of empirical neural loss landscapes is obtained, we will be able to derive theoretical results in more realistic settings than previous works, which will hopefully be able to guide the design of more efficient optimization algorithms. A few more steps in this direction are therefore necessary to lead to immediate algorithmic improvements.