Despite huge successes on a wide range of tasks, neural networks (NNs) are known to sometimes struggle to generalize to unseen data. Many approaches have been proposed over the years to promote the generalization ability of NNs, collectively known as regularization techniques. These are used as common practice under the assumption that any regularization added to the pipeline would result in a performance improvement. In this study, we investigate whether this assumption holds in practice. First, we provide a broad review of regularization techniques, including modern theories such as double descent. We propose a taxonomy of methods under four broad categories, namely: 1) data-based strategies; 2) architecture strategies; 3) training strategies; and 4) loss function strategies. Notably, we highlight the contradictions and correspondences between the approaches in these broad classes. Further, we perform an empirical comparison of the various regularization techniques on classification tasks for ten numerical and image datasets applied to the multilayer perceptron and convolutional NN architectures. Results show that the efficacy of regularization is dataset-dependent. For example, the use of a regularization term only improved performance on numeric datasets, whereas batch normalization improved performance on image datasets only. Generalization is crucial to machine learning; thus, understanding the effects of applying regularization techniques and considering the connections between them is essential to the appropriate use of these methods in practice.