Provably Correct Learning Algorithms in the Presence of Time-Varying Features Using a Variational Perspective

Features in machine learning problems are often time-varying and may be\nrelated to outputs in an algebraic or dynamical manner. The dynamic nature of\nthese machine learning problems renders current higher order accelerated\ngradient descent methods unstable or weakens their convergence guarantees.\nInspired by methods employed in adaptive control, this paper proposes new\nalgorithms for the case when time-varying features are present, and\ndemonstrates provable performance guarantees. In particular, we develop a\nunified variational perspective within a continuous time algorithm. This\nvariational perspective includes higher order learning concepts and\nnormalization, both of which stem from adaptive control, and allows stability\nto be established for dynamical machine learning problems where time-varying\nfeatures are present. These higher order algorithms are also examined for\nprovably correct learning in adaptive control and identification. Simulations\nare provided to verify the theoretical results.\n