In this paper, a general stochastic optimization procedure is studied,\nunifying several variants of the stochastic gradient descent such as, among\nothers, the stochastic heavy ball method, the Stochastic Nesterov Accelerated\nGradient algorithm (S-NAG), and the widely used Adam algorithm. The algorithm\nis seen as a noisy Euler discretization of a non-autonomous ordinary\ndifferential equation, recently introduced by Belotto da Silva and Gazeau,\nwhich is analyzed in depth. Assuming that the objective function is non-convex\nand differentiable, the stability and the almost sure convergence of the\niterates to the set of critical points are established. A noteworthy special\ncase is the convergence proof of S-NAG in a non-convex setting. Under some\nassumptions, the convergence rate is provided under the form of a Central Limit\nTheorem. Finally, the non-convergence of the algorithm to undesired critical\npoints, such as local maxima or saddle points, is established. Here, the main\ningredient is a new avoidance of traps result for non-autonomous settings,\nwhich is of independent interest.\n