Summary
This work combines variational perspective approach with high-resolution ODE functions to investigate the Nesterov accelerated gradient descent algorithm (NAG). With variational perspective, the authors reconstruct various high-resolution ODEs derived in previous research using alternative methods. Moreover, through this approach, they propose a special representation of NAG that exhibits an improved convergence rate in terms of gradient norm minimization. The authors also discuss some new properties of rate-matching technique. Finally, the authors analyze the stochastic setting both theoretically and empirically.
Strengths
1. The idea of combining variational perspective and high-resolution ODEs by including external forces is very interesting.
2. The authors show several important theoretical results in their manuscript. By carefully checking their proofs of theorems (except for Section 5 because of time), I think overall these results are correct.
3. The numerical results indicate the potential of having better optimization algorithms based on the theoretical results in this manuscript.
4. Overall, the manuscript is well written.
Overall, I have a positive impression of this work, however, I also admit that I am not an expert in this specific field, which may affect my confidence in assessing its accuracy and significance.
Weaknesses
1. It would be better to have a more detailed introduction to low-resolution ODEs and high-resolution ODEs. Without reading some previous work, it is difficult to understand the differences between low-resolution ODEs and high-resolution ODEs. Are high-resolution ODEs the ODE functions that contain the learning rate $s$? Why do people need to care about high-resolution ODEs?
2. I feel that the proof of Proposition 4.1 is more of an intuition rather than a rigorous demonstration. The treatment of the condition $s\to 0$ seems imprecise, as sometimes the authors will directly consider this condition as $s=0$ (e.g. line 60), while at other times the authors maintain $s$ to be a non-zero value (e.g. line 59). The use of the word “approximately” in Prop 4.1 is also vague.
3. Some typos/ unclear parts in the proofs that the authors may need to double-check.
(a) line 84, $\frac{\partial L}{\partial X}(X_t,...)$ should be $\frac{\partial L}{\partial X_t}(X_t,...)$.
(b) equation (7), left hand side, $\bigtriangledown f$ should be $\bigtriangledown f(X_t)$.
(c) Theorem 2.1, $\dot{\gamma}=e^{\alpha t}$ should be $\dot{\gamma}=e^{\alpha_t}$.
(d) line 138, ODE(14) should be ODE(12).
(e) equation (25) $(3/t+\sqrt{s}\bigtriangledown f(X_t))$ should be $(3/t+\sqrt{s}\bigtriangledown^2 f(X_t))$.
(f) line 196, $\sigma$ not introduced, is it the variance of the noise?
(g) equation (38) (appendix), second line $\sqrt(s)e^{-\alpha_t}\ddot{\beta_t}$ should be $\sqrt(s)e^{-2\alpha_t}\ddot{\beta_t}$.
(h) line 392, (7) should be (11).
(i) line 406, first equality $+\frac{1}{2}||v_k-x^*||^2$ should be $-\frac{1}{2}||v_k-x^*||^2$.
(j) line 408, the term $+\frac{s^2(k+2)}{4}||\bigtriangledown f(x_{k+1})||^2$ in the second inequality is left out, therefore the authors need to double-check whether the results still hold after considering this term.
(k) line 411, the first equaltiy redundant,
(l) line 413 "not ethat" typo
(m) line 421, the term $\frac{ks}{2}(\bigtriangledown f(x_k-\bigtriangledown f(x_k))$ miss ")".
(n) equation (62), $(3/t+\sqrt{s}\bigtriangledown f(X_t))$ should be $(3/t+\sqrt{s}\bigtriangledown^2 f(X_t))$.
4. Figure 1 is hard to read. (font size too small)
5. It would be better to describe what "NAG" means (Nesterov accelerated gradient?) the first time this abbreviation is used.
Questions
1. It is unclear to me that equation (21) and equation (27) (by replacing one term) is equal to NAG (line 76), why is that?
2. In line 211, the authors say that practically $k_0$ is lower than the term $(\cdot)^{1/\alpha}$. However, one of the conditions in Theorem 5.1 and Theorem 5.2 is that $k_0\geq (\cdot)^{1/\alpha}$, does that mean this condition will not be satisfied in practice?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
The authors don't specifically discuss the limitations of this work. The authors may consider adding a paragraph in their manuscript to discuss the limitations of their work based on the reviewer's feedback.
I don't think there will be a significant negative societal impact of this work.