Summary
This paper conducts a continuous-time analysis of an acceleration method called "anchoring", where the main contributions are four-fold. The authors provided a unified analysis of the convergence rate of anchor acceleration, which includes both the constant and adaptive cases. Then the authors presented an adaptive method for anchor acceleration that is inspired by our analysis and achieves faster convergence rates than the constant method. After this, the authors proved that the adaptive method is robust to noise and can handle non-convex optimization problems. Finally, the authors provided numerical experiments that demonstrate the effectiveness of the adaptive method on various optimization problems. Overall, the paper provides a valuable contribution to the field of optimization and is in general well-written and well-presented.
Strengths
This paper provides a comprehensive analysis of the convergence rate of anchor acceleration, which includes both the constant and adaptive cases. The paper provides a clear and concise presentation of both the theoretical analysis, where I checked the technical proofs of all results with a detailed focus on Theorem 3.1 [Section E.5] and Theorem 6.2 [Section H.2] and they are solid. The idea of using the factor $\frac{2 \mu}{e^{2 \mu t}-1}$ is is new and reasonable, since for \mu-strongly convex objectives $\mu\to 0^+$ it is consistent with the standard $\frac{1}{t}$ anchoring rate (the analysis provided is significantly more general, which should be honored). The adaptive method presented in the paper is also interesting, which appears faster convergence rates than the constant method and is robust to noise and non-convex optimization problems. The paper also clearly presented numerical experiments which demonstrate the effectiveness of the adaptive method on various optimization problems.
Weaknesses
The paper assumes a certain level of mathematical background, which might be difficult for some readers unfamiliar with literature to follow. Further, the paper seems not provide a comparison of the adaptive method with other state-of-the-art optimization methods. In addition, the paper does not provide a detailed discussion of the limitations of the adaptive method and areas for future research. I have not checked but the assumptions of the adaptive method presented in the paper might be more stringent than required.
Questions
Are there some mismatches in the definition of Lyapunov functions at various places? For instance, the $V(t)$ definition in Line 133 [Corollary 3.3], when pinning $\beta = \frac{1}{t}$, and the last in Line 147 [last but one display of Section 3.1] differs by a factor of 2. These are minor, but I do encourage the authors to check them carefully.
Can you provide examples of applications where anchor acceleration might be particularly useful? I understand that anchor acceleration has been discovered to be an acceleration mechanism for minimax optimization and fixed-point problems, but would anchor acceleration be useful in other optimization problems as well?
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
This is a theoretical paper and admits no negative social impacts, to my best knowledge.