Invertible neural networks (INNs) have been used to design generative models,\nimplement memory-saving gradient computation, and solve inverse problems. In\nthis work, we show that commonly-used INN architectures suffer from exploding\ninverses and are thus prone to becoming numerically non-invertible. Across a\nwide range of INN use-cases, we reveal failures including the non-applicability\nof the change-of-variables formula on in- and out-of-distribution (OOD) data,\nincorrect gradients for memory-saving backprop, and the inability to sample\nfrom normalizing flow models. We further derive bi-Lipschitz properties of\natomic building blocks of common architectures. These insights into the\nstability of INNs then provide ways forward to remedy these failures. For tasks\nwhere local invertibility is sufficient, like memory-saving backprop, we\npropose a flexible and efficient regularizer. For problems where global\ninvertibility is necessary, such as applying normalizing flows on OOD data, we\nshow the importance of designing stable INN building blocks.\n