Set prediction is about learning to predict a collection of unordered\nvariables with unknown interrelations. Training such models with set losses\nimposes the structure of a metric space over sets. We focus on stochastic and\nunderdefined cases, where an incorrectly chosen loss function leads to\nimplausible predictions. Example tasks include conditional point-cloud\nreconstruction and predicting future states of molecules. In this paper, we\npropose an alternative to training via set losses by viewing learning as\nconditional density estimation. Our learning framework fits deep energy-based\nmodels and approximates the intractable likelihood with gradient-guided\nsampling. Furthermore, we propose a stochastically augmented prediction\nalgorithm that enables multiple predictions, reflecting the possible variations\nin the target set. We empirically demonstrate on a variety of datasets the\ncapability to learn multi-modal densities and produce different plausible\npredictions. Our approach is competitive with previous set prediction models on\nstandard benchmarks. More importantly, it extends the family of addressable\ntasks beyond those that have unambiguous predictions.\n