Pooling is neither necessary nor sufficient for appropriate deformation stability in CNNs

Many of our core assumptions about how neural networks operate remain\nempirically untested. One common assumption is that convolutional neural\nnetworks need to be stable to small translations and deformations to solve\nimage recognition tasks. For many years, this stability was baked into CNN\narchitectures by incorporating interleaved pooling layers. Recently, however,\ninterleaved pooling has largely been abandoned. This raises a number of\nquestions: Are our intuitions about deformation stability right at all? Is it\nimportant? Is pooling necessary for deformation invariance? If not, how is\ndeformation invariance achieved in its absence? In this work, we rigorously\ntest these questions, and find that deformation stability in convolutional\nnetworks is more nuanced than it first appears: (1) Deformation invariance is\nnot a binary property, but rather that different tasks require different\ndegrees of deformation stability at different layers. (2) Deformation stability\nis not a fixed property of a network and is heavily adjusted over the course of\ntraining, largely through the smoothness of the convolutional filters. (3)\nInterleaved pooling layers are neither necessary nor sufficient for achieving\nthe optimal form of deformation stability for natural image classification. (4)\nPooling confers too much deformation stability for image classification at\ninitialization, and during training, networks have to learn to counteract this\ninductive bias. Together, these findings provide new insights into the role of\ninterleaved pooling and deformation invariance in CNNs, and demonstrate the\nimportance of rigorous empirical testing of even our most basic assumptions\nabout the working of neural networks.\n