In deep neural networks, the spectral norm of the Jacobian of a layer bounds\nthe factor by which the norm of a signal changes during forward/backward\npropagation. Spectral norm regularizations have been shown to improve\ngeneralization, robustness and optimization of deep learning methods. Existing\nmethods to compute the spectral norm of convolution layers either rely on\nheuristics that are efficient in computation but lack guarantees or are\ntheoretically-sound but computationally expensive. In this work, we obtain the\nbest of both worlds by deriving {\\it four} provable upper bounds on the\nspectral norm of a standard 2D multi-channel convolution layer. These bounds\nare differentiable and can be computed efficiently during training with\nnegligible overhead. One of these bounds is in fact the popular heuristic\nmethod of Miyato et al. (multiplied by a constant factor depending on filter\nsizes). Each of these four bounds can achieve the tightest gap depending on\nconvolution filters. Thus, we propose to use the minimum of these four bounds\nas a tight, differentiable and efficient upper bound on the spectral norm of\nconvolution layers. We show that our spectral bound is an effective regularizer\nand can be used to bound either the lipschitz constant or curvature values\n(eigenvalues of the Hessian) of neural networks. Through experiments on MNIST\nand CIFAR-10, we demonstrate the effectiveness of our spectral bound in\nimproving generalization and provable robustness of deep networks.\n