A major challenge in training deep neural networks is <italic>overfitting</italic>, i.e. inferior performance on unseen test examples compared to performance on training examples. To reduce overfitting, stochastic regularization methods have shown superior performance compared to deterministic weight penalties on a number of image recognition tasks. Stochastic methods, such as Dropout and Shakeout, in expectation, are equivalent to imposing a ridge and elastic-net penalty on the model parameters, respectively. However, the choice of the norm of the weight penalty is problem dependent and is not restricted to <inline-formula> <tex-math notation="LaTeX">$\{L_{1},L_{2}\}$ </tex-math></inline-formula>. Therefore, in this paper, we propose the Bridgeout stochastic regularization technique and prove that it is equivalent to an <inline-formula> <tex-math notation="LaTeX">$L_{q}$ </tex-math></inline-formula> penalty on the weights, where the norm <inline-formula> <tex-math notation="LaTeX">$q$ </tex-math></inline-formula> can be learned as a hyperparameter from data. Experimental results show that Bridgeout results in sparse model weights, improved gradients, and superior classification performance compared with Dropout and Shakeout on synthetic and real data sets.