Summary
This paper presents a 2-stage generative model for text. The first stage involves training a VAE over the data to obtain an effective encoder and decoder modules. The second stage uses the output of the frozen VAE encoder as the hidden state of the input text and learns a regular continuous diffusion model over the encoded hidden states of training data instances. For generation, the diffusion model produces a hidden state from noise and this hidden state is fed into the VAE decoder to generate text. This model is evaluated on sentiment-guided generation, text completion, and summarization and compared against baselines that include token-based diffusion models (instead of hidden state text diffusion models in this work), VAE/Enc-dec models, and standard autoregressive models (like fine tuned GPT-2).
Strengths
– Although the approach seems straightforward, to the best of my knowledge, surprisingly I am not aware of other latent sequence/paragraph embedding based diffusion models for text generation. Hence, this fills a gap in the research on diffusion models for text.
– The experimental design is reasonable and informative.
– This approach seems to be more fluent, diverse, and effective than popular token-based diffusion models following the results.
– Although typically less fluent than the autoregressive models, it still performs competitively when considering other metrics like diversity, reference overlap, and control.
Weaknesses
– I have some concerns about evaluation, particularly related to the quality of baseline models. For example, on CNN-DM summarization task, publicly available results (https://paperswithcode.com/sota/abstractive-text-summarization-on-cnn-daily) show that T5 achieve R-L that is ~7-10 points better than reported in this paper (and is hence better than the R-L proposed model achieves). Therefore, I am doubtful about the quality of training/tuning of the baseline models reported in the paper.
– Choosing “first k hidden state vectors” from the encoder in the VAE seems arbitrary. What is the motivation behind this hyperparameter? What is the effect of this k?
– The paper alludes to learning “paragraph embeddings” which capture high level semantic properties which enables “coarse-to-fine” generation and allows for “planning”. However, no evidence is presented in this paper about compositional/planning capabilities of the model. While it is true that paragrpahs/tokens are generated from a lower dimensional manifold, it doesn’t automatically imply that these embeddings are linguistically interesting or allow for controlled planning-oriented generation. At an abstract conceptual level, how different are these representations from VAE based representation for example?
– Although this is a generative model, the training objective is arbitrary and doesn’t seem linked to maximizing the likelihood/ learning the distribution of the training data. This is because the VAE and the diffusion components are separately trained and hence preclude a clean interpretation of the training objective.
– I am not convinced that the “Distributional smoothness metric” reflect the nature of the manifold as is claimed in the paper.
– Runtime analysis seems to compare unbatched autoregressive models with batched diffusion models. Sorting by length and then padding for batching is a standard practice for autoregressive models.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.