A Unified Theory of Early Visual Representations from Retina to Cortex through Anatomically Constrained Deep CNNs

The visual system is hierarchically organized to process visual information\nin successive stages. Neural representations vary drastically across the first\nstages of visual processing: at the output of the retina, ganglion cell\nreceptive fields (RFs) exhibit a clear antagonistic center-surround structure,\nwhereas in the primary visual cortex, typical RFs are sharply tuned to a\nprecise orientation. There is currently no unified theory explaining these\ndifferences in representations across layers. Here, using a deep convolutional\nneural network trained on image recognition as a model of the visual system, we\nshow that such differences in representation can emerge as a direct consequence\nof different neural resource constraints on the retinal and cortical networks,\nand we find a single model from which both geometries spontaneously emerge at\nthe appropriate stages of visual processing. The key constraint is a reduced\nnumber of neurons at the retinal output, consistent with the anatomy of the\noptic nerve as a stringent bottleneck. Second, we find that, for simple\ncortical networks, visual representations at the retinal output emerge as\nnonlinear and lossy feature detectors, whereas they emerge as linear and\nfaithful encoders of the visual scene for more complex cortices. This result\npredicts that the retinas of small vertebrates should perform sophisticated\nnonlinear computations, extracting features directly relevant to behavior,\nwhereas retinas of large animals such as primates should mostly encode the\nvisual scene linearly and respond to a much broader range of stimuli. These\npredictions could reconcile the two seemingly incompatible views of the retina\nas either performing feature extraction or efficient coding of natural scenes,\nby suggesting that all vertebrates lie on a spectrum between these two\nobjectives, depending on the degree of neural resources allocated to their\nvisual system.\n