We develop a general framework based on splines to understand the interpolation properties of overparameterized neural networks. We prove that minimum "norm" two-layer neural networks (with appropriately chosen activation functions) that interpolate scattered data are minimal knot splines. Our results follow from understanding key relationships between notions of neural network "norms", linear operators, and continuous-domain linear inverse problems.