Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence

We investigate the reasons for the performance degradation incurred with\nbatch-independent normalization. We find that the prototypical techniques of\nlayer normalization and instance normalization both induce the appearance of\nfailure modes in the neural network's pre-activations: (i) layer normalization\ninduces a collapse towards channel-wise constant functions; (ii) instance\nnormalization induces a lack of variability in instance statistics, symptomatic\nof an alteration of the expressivity. To alleviate failure mode (i) without\naggravating failure mode (ii), we introduce the technique "Proxy Normalization"\nthat normalizes post-activations using a proxy distribution. When combined with\nlayer normalization or group normalization, this batch-independent\nnormalization emulates batch normalization's behavior and consistently matches\nor exceeds its performance.\n