In this paper, we consider both first- and second-order techniques to address\ncontinuous optimization problems arising in machine learning. In the\nfirst-order case, we propose a framework of transition from deterministic or\nsemi-deterministic to stochastic quadratic regularization methods. We leverage\nthe two-phase nature of stochastic optimization to propose a novel first-order\nalgorithm with adaptive sampling and adaptive step size. In the second-order\ncase, we propose a novel stochastic damped L-BFGS method that improves on\nprevious algorithms in the highly nonconvex context of deep learning. Both\nalgorithms are evaluated on well-known deep learning datasets and exhibit\npromising performance.\n