Storage of Input Values Within Core of Neural Network Inference Circuit

Some embodiments provide a neural network inference circuit (NNIC) for executing a NN that includes multiple computation nodes at multiple layers. Each of a set of the computation nodes includes a dot product of input values and weight values. The NNIC includes a set of dot product cores, each of which includes (i) partial dot product computation circuits to compute dot products between input values and weight values and (ii) memories to store the sets of weight values and sets of input values for a layer of the neural network. The input values for a particular layer are arranged in a plurality of two-dimensional grids. A particular core stores all of the input values of a subset of the two-dimensional grids. Input values having a same set of coordinates in each respective grid of the subset of the two-dimensional grids are stored sequentially within the memories of the particular core.