Inspired by recent advances in the field of expert-based approximations of\nGaussian processes (GPs), we present an expert-based approach to large-scale\nmulti-output regression using single-output GP experts. Employing a deeply\nstructured mixture of single-output GPs encoded via a probabilistic circuit\nallows us to capture correlations between multiple output dimensions\naccurately. By recursively partitioning the covariate space and the output\nspace, posterior inference in our model reduces to inference on single-output\nGP experts, which only need to be conditioned on a small subset of the\nobservations. We show that inference can be performed exactly and efficiently\nin our model, that it can capture correlations between output dimensions and,\nhence, often outperforms approaches that do not incorporate inter-output\ncorrelations, as demonstrated on several data sets in terms of the negative log\npredictive density.\n