We propose to focus on the problem of discovering neural network\narchitectures efficient in terms of both prediction quality and cost. For\ninstance, our approach is able to solve the following tasks: learn a neural\nnetwork able to predict well in less than 100 milliseconds or learn an\nefficient model that fits in a 50 Mb memory. Our contribution is a novel family\nof models called Budgeted Super Networks (BSN). They are learned using gradient\ndescent techniques applied on a budgeted learning objective function which\nintegrates a maximum authorized cost, while making no assumption on the nature\nof this cost. We present a set of experiments on computer vision problems and\nanalyze the ability of our technique to deal with three different costs: the\ncomputation cost, the memory consumption cost and a distributed computation\ncost. We particularly show that our model can discover neural network\narchitectures that have a better accuracy than the ResNet and Convolutional\nNeural Fabrics architectures on CIFAR-10 and CIFAR-100, at a lower cost.\n