The performance of gradient-based optimization strategies depends heavily on\nthe initial weights of the parametric model. Recent works show that there exist\nweight initializations from which optimization procedures can find the\ntask-specific parameters faster than from uniformly random initializations and\nthat such a weight initialization can be learned by optimizing a specific model\narchitecture across similar tasks via MAML (Model-Agnostic Meta-Learning).\nCurrent methods are limited to populations of classification tasks that share\nthe same number of classes due to the static model architectures used during\nmeta-learning. In this paper, we present HIDRA, a meta-learning approach that\nenables training and evaluating across tasks with any number of target\nvariables. We show that Model-Agnostic Meta-Learning trains a distribution for\nall the neurons in the output layer and a specific weight initialization for\nthe ones in the hidden layers. HIDRA explores this by learning one master\nneuron, which is used to initialize any number of output neurons for a new\ntask. Extensive experiments on the Miniimagenet and Omniglot data sets\ndemonstrate that HIDRA improves over standard approaches while generalizing to\ntasks with any number of target variables. Moreover, our approach is shown to\nrobustify low-capacity models in learning across complex tasks with a high\nnumber of classes for which regular MAML fails to learn any feasible\ninitialization.\n