Latent Diffusion for Language Generation

Summary

This work proposes a new type of diffusion language model based on latent representations. The proposed approach leverages the continuous representation space produced by pre-trained encoder-decoder language models (BART and FLAN-T5). To embed the variable-length encoder features, the author(s) propose to adopt Perceiver Resampler to construct a fixed length latent representation. After obtaining the diffused representation, a reconstruction network is used whose output is later taken by the frozen language decoder to generate the output. Extensive experiments are performed on a few types of tasks across different benchmarks. The experimental results verify that the proposed approach outperforms existing diffusion-based methods and performs comparably with directly fine-tuned language models. Good discussions on future work are included in this work.

Strengths

* The paper is generally well-written and the presentation is clear. * The proposed model is novel in terms of architectural design and motivations. * Extensive experiments are conducted to sufficiently evaluate the method and the results are convincing.

Weaknesses

* One major weakness of this work is its potential impact to the research community in the long-term. From the presented results (Table 6), simply fine-tuning the language model still yields the best results across the board. In addition, the existing autoregressive language model such as ChatGPT and GPT-4 have proven their exceptional capabilities in various applications. On the other hand, it is hard to judge the performance of diffusion-based approaches, especially when the model is scaled up. I believe scaling up the proposed model is out-of-scope of the this work, but it worth considered by the author(s). * **An important comparison is missing**. From the first impression, the encoding of language encoder and the sequential decoding of language decoder are shared by both the proposed method and the standard autoregressive language model. The additional diffusion process would inevitably increase the inference latency. While the author(s) discuss the downside of inference latency in Section 6, it is important to compare both approaches systematically.

Questions

1. Could you elaborate more details of the language autoencoder experiments in Section 5.1? Do you train the compression and reconstruction networks to let the model replicate the input in its output? 2. In line 268, why do you need to specify the length for inference. I think the generation length is determined by the emission of EOS token of the language decoder. Please correct me if I am wrong.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please see the sections of weaknesses and questions.

Summary

The paper proposes a latent diffusion model for text generation, by stacking pre-trained powerful language models and fine-tuned projection layers and diffusion networks together. The input is first encoded into hiddens, then the diffusion model generates the hidden used as the condition of the language decoder. The final results are generated then. Experiments are conducted on both conditional and unconditional language generation tasks.

Strengths

- The paper is well-written and easy to follow. - The method part is in detail which makes readers understand the model immediately.

Weaknesses

- The contribution is weak. 1) the diffusion model is only utilized to generate the condition of the LM decoder, which means that the model just provides another projection of the original LLM encoder output. Why is this step necessary? As a cost, the generation speed is very slow. I saw in the appendix that generating 1k samples needs 3 minutes. What will the original BART cost? 2) The performance of the model heavily depends on the backbone LM. For example, BART outperforms FLAN-T5 by a large margin. What if training from scratch? What is the performance with other architectures? What if the performance of backbones varies from task, for example, in some tasks BART is better but in others FLAN-T5 is better? This kind of dependence strictly limits the generality of the model. - The experiments lack some new baselines, such as GENIE, Difformer, etc. The choice of datasets is also wired, why not conduct experiments on exactly the same ones as Diffuseq?

Questions

Please answer the questions in Weaknesses. What are the purpose and benefits of utilizing diffusion in such a framework?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors have discussed limitations.

Summary

The paper demonstrates that encoder-decoder language models can be used to efficiently learn high-quality language autoencoders and to learn continuous diffusion models in the latent space of language autoencoders so that continuous latent representations of natural language can be sampled using pre-trained decoders.

Strengths

1. The paper proposes an additional compression module which maps high-dimensional encoder representations to low-dimensional fixed-length representations. And the corresponding reconstruction network is trained to map these fixed-length features back to high-dimensional features, guiding the frozen language decoder to reconstruct the original language. 2. The paper verifies the effectiveness of the model's unconditional, class-conditional, and sequence-to-sequence language generation methods on several different datasets.

Weaknesses

1. The motivation of the paper is somewhat weak. Why do we need to apply the latent diffusion model on text generation? What are the advantages over autoregressive and non-autoregressive models? It seems that the performance (MBR-5 in Table 6) is worse than the original AR model. Maybe it is faster? Overall, the lack of clear motivation makes the paper read like a technical report. 2. I have reservations about the ability of the variational paragraph embedder to learn effective representations of sentences with different lengths. What would happen if there is a significant difference in sentence lengths, such as one being very short (e.g., 5 tokens) and the other being much longer (e.g., 512 tokens or more)? Moreover, what is the influence of fixed length number, $l$ in line 102?

Questions

1. Is the autoencoder joint training with the denoising network? 2. What is the influence of different VAE models on the final generation performance?

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

Yes, the authors clearly give the limitations of their work.

Summary

This paper introduces a novel approach to learning diffusion models in the latent space of an encoder-decoder language autoencoder. The proposed latent diffusion model utilizes continuous latent codes to generate text through sampling, leveraging a pre-trained decoder. The effectiveness of this approach is demonstrated across various unconditional and conditional sequence generation tasks.

Strengths

1. The idea presented in the paper is straightforward and easy to understand. It extends the concept of latent diffusion models (LDM) used in other domains, such as images, to the language domain, showcasing the versatility of this approach. 2. This paper stands out by incorporating an autoregressive decoder into diffusion models, which combines the strengths of both methods. This integration significantly enhances the quality of text generation, distinguishing it from existing diffusion-based language models.

Weaknesses

The basic idea of learning diffusion models in the latent space has been well explored in the image domain. Adding a self-conditioning trick has also been used in previous diffusion-LM literatures. Therefore the overall novelty is relatively limited.

Questions

1. In line 107, if the latent vector is normalized by its norm, then the latents are not Gaussian anymore but become a point on the hypersphere. Will it make more sense to divide the latent with its standard deviation estimated from the data? 2. From the encoder to latents and the latents to the decoder involve two cross-attention, will this cause training instability when optimizing things end to end? On the other hand, how does the model avoid copying all encoder information to the target side? 3. How are the length and dimensions of the latents decided? Will it be possible to design variable lengths of latents to account for the actual sequence length?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations were addressed.

Summary

This paper introduces a latent diffusion model for natural language generation, utilizing a continuous diffusion framework. The authors leverage pretrained encoder-decoder models, such as BART and Flan-T5, to establish a continuous feature space. They propose a Perceiver Resampler as a compression module to handle variable-length and high-dimensional challenges, allowing effective learning of the diffusion process and parameterized denoising. The generative process includes a reconstruction module to map the low-dimensional latent space back to the feature space and is supported in unconditional, class-conditional, and seq2seq settings. The proposed approach demonstrates its effectiveness through extensive experiments.

Strengths

1. The utilization of pretrained encoder-decoder models to establish a feature space is a clever and efficient approach, distinguishing it from previous work that relied on less-capable embedding "encoder-decoder" for latent diffusion. 2. The paper conducts extensive experiments in various settings, including unconditional, class-conditional, and seq-conditional text generation. The evaluation metrics cover aspects of quality, diversity, and other relevant factors.

Weaknesses

1. Evaluation concerns arise from the lack of a multilingual setting in the benchmark datasets investigated. The benchmark datasets investigated in the paper were various, which was good, but lacked of multilingual setting, making the evaluation not fully convincing. Several previous diffusion-based text generation studies experimented on machine translation, e.g, CDCD [2], Difformer [2], and DINOISER [3] studied on WMT14 English-to-German dataset, which is most commonly-used setting for non-autoregressive text generation. Can the proposed LD4LG model also be applied in a multilingual setting? 2. The paper lacks a comprehensive literature review, hence comparison and discussion on its contributions with relevant studies. In the experiments, only diffusion-lm and diffuseq were compared. Additionally, the related work section primarily cites papers from 2022, while significant progress has been made in text diffusion research this year. What are the advantages of LD4LG compared to other continuous diffusion models, such as CDCD, Difformer, and DINOISER? How does it compare to discrete diffusion models like DiffusionBERT, reparameterized discrete diffusion model (RDM), and DiffusER? --- [1] Continuous diffusion for categorical data [2] Difformer: Empowering Diffusion Model on Embedding Space for Text Generation [3] DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises [4] DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models [5] A Reparameterized Discrete Diffusion Model for Text Generation [6] DiffusER: Discrete Diffusion via Edit-based Reconstruction

Questions

See weaknesses. I am happy to raise my rating upon the authors' responses to my concerns.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

n/a

Thank you for your comprehensive response

Thank you for the rigorous replies. I appreciate the author(s)' honest acknowledgments of this work's limitations. I have raised my score to 6.

Thanks for your response and increase my rating

Dear authors, Thank you so much for your efforts in providing thorough results and discussions! Although there're still rooms for improvement for multilingual/machine translation given the WMT14 De-En results, the responses have mostly addressed my concerns in general. I accordingly raise my rating from 4 to 5. Please add the new results and discussions in the revision.