Reverse engineering learned optimizers reveals known and novel mechanisms

Learned optimizers are algorithms that can themselves be trained to solve\noptimization problems. In contrast to baseline optimizers (such as momentum or\nAdam) that use simple update rules derived from theoretical principles, learned\noptimizers use flexible, high-dimensional, nonlinear parameterizations.\nAlthough this can lead to better performance in certain settings, their inner\nworkings remain a mystery. How is a learned optimizer able to outperform a well\ntuned baseline? Has it learned a sophisticated combination of existing\noptimization techniques, or is it implementing completely new behavior? In this\nwork, we address these questions by careful analysis and visualization of\nlearned optimizers. We study learned optimizers trained from scratch on three\ndisparate tasks, and discover that they have learned interpretable mechanisms,\nincluding: momentum, gradient clipping, learning rate schedules, and a new form\nof learning rate adaptation. Moreover, we show how the dynamics of learned\noptimizers enables these behaviors. Our results help elucidate the previously\nmurky understanding of how learned optimizers work, and establish tools for\ninterpreting future learned optimizers.\n