Recent work on Renyi Differential Privacy has shown the feasibility of\napplying differential privacy to deep learning tasks. Despite their promise,\nhowever, differentially private deep networks often lag far behind their\nnon-private counterparts in accuracy, showing the need for more research in\nmodel architectures, optimizers, etc. One of the barriers to this expanded\nresearch is the training time -- often orders of magnitude larger than training\nnon-private networks. The reason for this slowdown is a crucial privacy-related\nstep called "per-example gradient clipping" whose naive implementation undoes\nthe benefits of batch training with GPUs. By analyzing the back-propagation\nequations we derive new methods for per-example gradient clipping that are\ncompatible with auto-differentiation (e.g., in PyTorch and TensorFlow) and\nprovide better GPU utilization. Our implementation in PyTorch showed\nsignificant training speed-ups (by factors of 54x - 94x for training various\nmodels with batch sizes of 128). These techniques work for a variety of\narchitectural choices including convolutional layers, recurrent networks,\nattention, residual blocks, etc.\n