While recent work has shown that scores from models trained by the ubiquitous\nmasked language modeling (MLM) objective effectively discriminate probable from\nimprobable sequences, it is still an open question if these MLMs specify a\nprincipled probability distribution over the space of possible sequences. In\nthis paper, we interpret MLMs as energy-based sequence models and propose two\nenergy parametrizations derivable from the trained MLMs. In order to draw\nsamples correctly from these models, we develop a tractable sampling scheme\nbased on the Metropolis--Hastings Monte Carlo algorithm. In our approach,\nsamples are proposed from the same masked conditionals used for training the\nmasked language models, and they are accepted or rejected based on their energy\nvalues according to the target distribution. We validate the effectiveness of\nthe proposed parametrizations by exploring the quality of samples drawn from\nthese energy-based models for both open-ended unconditional generation and a\nconditional generation task of machine translation. We theoretically and\nempirically justify our sampling algorithm by showing that the masked\nconditionals on their own do not yield a Markov chain whose stationary\ndistribution is that of our target distribution, and our approach generates\nhigher quality samples than other recently proposed undirected generation\napproaches (Wang et al., 2019, Ghazvininejad et al., 2019).\n