Denoising diffusion probabilistic models (DDPMs) have shown impressive\nresults on sequence generation by iteratively corrupting each example and then\nlearning to map corrupted versions back to the original. However, previous work\nhas largely focused on in-place corruption, adding noise to each pixel or token\nindividually while keeping their locations the same. In this work, we consider\na broader class of corruption processes and denoising models over sequence data\nthat can insert and delete elements, while still being efficient to train and\nsample from. We demonstrate that these models outperform standard in-place\nmodels on an arithmetic sequence task, and that when trained on the text8\ndataset they can be used to fix spelling errors without any fine-tuning.\n