Despite the fact that the loss functions of deep neural networks are highly\nnon-convex, gradient-based optimization algorithms converge to approximately\nthe same performance from many random initial points. One thread of work has\nfocused on explaining this phenomenon by characterizing the local curvature\nnear critical points of the loss function, where the gradients are near zero,\nand demonstrating that neural network losses enjoy a no-bad-local-minima\nproperty and an abundance of saddle points. We report here that the methods\nused to find these putative critical points suffer from a bad local minima\nproblem of their own: they often converge to or pass through regions where the\ngradient norm has a stationary point. We call these gradient-flat regions,\nsince they arise when the gradient is approximately in the kernel of the\nHessian, such that the loss is locally approximately linear, or flat, in the\ndirection of the gradient. We describe how the presence of these regions\nnecessitates care in both interpreting past results that claimed to find\ncritical points of neural network losses and in designing second-order methods\nfor optimizing neural networks.\n