Gradient flows and proximal splitting methods: A unified view on accelerated and stochastic optimization

Proximal algorithms are well-suited for nonsmooth and constrained large-scale optimization problems and therefore suitable for applications in many areas of science. There are essentially four proximal algorithms based on fixed-point iterations currently known: forward-backward splitting, forward-backward-forward or Tseng splitting, Douglas-Rachford, and the Davis-Yin three operator splitting. In addition, the alternating direction method of multipliers (ADMM) is also closely related. In this paper we show that all of these algorithms can be derived as different discretizations of a single differential equation, namely the simple gradient flow. This is achieved through splitting methods for differential equations. Moreover, employing similar discretization schemes to a particular second-order differential equation, which we refer to as the accelerated gradient flow, results in accelerated variants of each respective proximal algorithm; we simultaneously consider two types of acceleration, although other choices are also possible. For instance, we propose accelerated variants of Davis-Yin and Tseng splitting, as well as accelerated extensions of ADMM. Interestingly, we show that ADMM and its accelerated variants correspond to rebalanced splittings, which is a recent technique designed to preserve steady states of the underlying differential equation. We show that all derived algorithms are valid first-order integrators under suitable assumptions. Our results strengthen the connections between optimization and continuous dynamical systems, offer a unified perspective on accelerated algorithms, and provide new accelerated algorithms.