This paper presents an empirical study on the weights of neural networks,\nwhere we interpret each model as a point in a high-dimensional space -- the\nneural weight space. To explore the complex structure of this space, we sample\nfrom a diverse selection of training variations (dataset, optimization\nprocedure, architecture, etc.) of neural network classifiers, and train a large\nnumber of models to represent the weight space. Then, we use a machine learning\napproach for analyzing and extracting information from this space. Most\ncentrally, we train a number of novel deep meta-classifiers with the objective\nof classifying different properties of the training setup by identifying their\nfootprints in the weight space. Thus, the meta-classifiers probe for patterns\ninduced by hyper-parameters, so that we can quantify how much, where, and when\nthese are encoded through the optimization process. This provides a novel and\ncomplementary view for explainable AI, and we show how meta-classifiers can\nreveal a great deal of information about the training setup and optimization,\nby only considering a small subset of randomly selected consecutive weights. To\npromote further research on the weight space, we release the neural weight\nspace (NWS) dataset -- a collection of 320K weight snapshots from 16K\nindividually trained deep neural networks.\n