Catastrophic forgetting undermines the effectiveness of deep neural networks\n(DNNs) in scenarios such as continual learning and lifelong learning. While\nseveral methods have been proposed to tackle this problem, there is limited\nwork explaining why these methods work well. This paper has the goal of better\nexplaining a popularly used technique for avoiding catastrophic forgetting:\nquadratic regularization. We show that quadratic regularizers prevent\nforgetting of past tasks by interpolating current and previous values of model\nparameters at every training iteration. Over multiple training iterations, this\ninterpolation operation reduces the learning rates of more important model\nparameters, thereby minimizing their movement. Our analysis also reveals two\ndrawbacks of quadratic regularization: (a) dependence of parameter\ninterpolation on training hyperparameters, which often leads to training\ninstability and (b) assignment of lower importance to deeper layers, which are\ngenerally the place forgetting occurs in DNNs. Via a simple modification to the\norder of operations, we show these drawbacks can be easily avoided, resulting\nin 6.2\\% higher average accuracy at 4.5\\% lower average forgetting. We confirm\nthe robustness of our results by training over 2000 models in different\nsettings. Code available at \\url{https://github.com/EkdeepSLubana/QRforgetting}\n