Most popular optimizers for deep learning can be broadly categorized as\nadaptive methods (e.g. Adam) and accelerated schemes (e.g. stochastic gradient\ndescent (SGD) with momentum). For many models such as convolutional neural\nnetworks (CNNs), adaptive methods typically converge faster but generalize\nworse compared to SGD; for complex settings such as generative adversarial\nnetworks (GANs), adaptive methods are typically the default because of their\nstability.We propose AdaBelief to simultaneously achieve three goals: fast\nconvergence as in adaptive methods, good generalization as in SGD, and training\nstability. The intuition for AdaBelief is to adapt the stepsize according to\nthe "belief" in the current gradient direction. Viewing the exponential moving\naverage (EMA) of the noisy gradient as the prediction of the gradient at the\nnext time step, if the observed gradient greatly deviates from the prediction,\nwe distrust the current observation and take a small step; if the observed\ngradient is close to the prediction, we trust it and take a large step. We\nvalidate AdaBelief in extensive experiments, showing that it outperforms other\nmethods with fast convergence and high accuracy on image classification and\nlanguage modeling. Specifically, on ImageNet, AdaBelief achieves comparable\naccuracy to SGD. Furthermore, in the training of a GAN on Cifar10, AdaBelief\ndemonstrates high stability and improves the quality of generated samples\ncompared to a well-tuned Adam optimizer. Code is available at\nhttps://github.com/juntang-zhuang/Adabelief-Optimizer\n