The visual world is vast and varied, but its variations divide into\nstructured and unstructured factors. We compose free-form filters and\nstructured Gaussian filters, optimized end-to-end, to factorize deep\nrepresentations and learn both local features and their degree of locality. Our\nsemi-structured composition is strictly more expressive than free-form\nfiltering, and changes in its structured parameters would require changes in\nfree-form architecture. In effect this optimizes over receptive field size and\nshape, tuning locality to the data and task. Dynamic inference, in which the\nGaussian structure varies with the input, adapts receptive field size to\ncompensate for local scale variation. Optimizing receptive field size improves\nsemantic segmentation accuracy on Cityscapes by 1-2 points for strong dilated\nand skip architectures and by up to 10 points for suboptimal designs. Adapting\nreceptive fields by dynamic Gaussian structure further improves results,\nequaling the accuracy of free-form deformation while improving efficiency.\n