AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation

Summary

This paper presents a diffusion model on text generation. The idea is generally interesting. It learns to diffuse sentence-level and token-level diffusion, where the latter one is diffused with dynamic movement speeds.Its experiments are well-designed and its empirical results are strong.

Strengths

1. The method is interesting with a perspective to discuss autoregressive and non-autoregressive diffusion models. 2. The skipping mechanism is useful to accelerate the generation process. 3. The empirical results are strong.

Weaknesses

1. The paper is hard to follow, and the writing should be improved. 2. Limited diiffusion model baseline and missing some related baselines.

Questions

1. I wonder if the skipping mechanism can also be applied to other diffusion models. 2. In Algo2 line 6, where does n come from? Do you enumerate n? 3. In Diffusion-LM, there is a rounding operation as Algo 2 line 9 at each diffusion step. It seems AR-Diffusion do not requires such rounding operation at each step. If true, would DPM-Solver help accelerating the diffusion process? 4. GENIE is the main (almost the only) diffusion model baseline in the experiments, but some related baselines are mssing, such as DiffuSeq [1], CDCD[2], Difformer [3], DINOISER[4] and RDM[5]. It seems the authors already noticed some of them in the related work wonder, and I think it would be helpful to include more complete baselines. If my conerns are addressed, I'm willing to raise my score. References 1. Gong et al. Diffuseq: Sequence to sequence text generation with diffusion models 2022 2. Dieleman et al. Continuous diffusion for categorical data 2022 3. Gao et al. Difformer: Empowering Diffusion Models on the Embedding Space for Text Generation, 2022 4. Ye et al. DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises 2023 5. Zheng et al. Reparameterized Discrete Diffusion for Text Generation 2023

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

2 fair

Contribution

3 good

Limitations

Yes

Summary

This paper presents AR-DIFFUSION, a diffusion model that displays auto-regression-like generation behavior. The primary contributions of this work can be summarized as follows: 1) A multi-level diffusion strategy is proposed, encompassing both sentence-level and token-level diffusion. 2) A skipping mechanism is introduced, which works in tandem with the multi-level diffusion strategy to expedite the process. 3) The superiority of the model over existing diffusion language models is verified in terms of text generation tasks and inference efficiency.

Strengths

The author presents an approach for integrating an auto-regressive-like mechanism into the diffusion model for text generation and has conducted comprehensive experiments to validate the efficacy of the proposed method. The idea of incorporating autoregressive dependency into the diffusion model is captivating.

Weaknesses

I find the experiments in this paper insufficiently convincing. My primary concern is that the main gains appear to be derived from MBR. With the skipping mechanism, the per-sentence generation step is reduced from 2000 to 20. This gives you chances to use K=500 for MBR, which is really large, as MBR is N^2 in computation. The NFE metric is somewhat misleading, as it only considers the number of model forwards and does not account for the MBR process. It would be more appropriate to report the runtime speed of all methods for a fair comparison. The comparison with baselines is not exhaustive. To substantiate the claim that their method benefits from introducing auto-regressive dependency back into text diffusion models, the authors should compare additional diffusion baselines, such as DIFFUSEQ and SeqDiffuSeq, with the same diffusion steps and candidates used in MBR, rather than just GENIE. Moreover, GENIE itself reported a Rouge-L score of 41.2 on CNN/Daily Mail, while the table in this paper shows only 32.1. This discrepancy should be clarified in the main text. If the lower score is due to a smaller parameter size, why not increase it to match GENIE's level? If the proposed method is effective, it could potentially be comparable to models like BART. As it stands, the parameter size of AR-Diffusion is too small to demonstrate its superiority. Furthermore, the comparison with NAR methods is not solid enough. Baselines such as CMLM and LevT were proposed three years ago. More recent methods like DA-Transformer should be included for a more comprehensive comparison.

Questions

Please answer the questions mentioned in the Weaknesses section.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

NA

Summary

This paper introduces a diffusion method optimized for the autoregressive text generation scheme. They employ different movement speeds for denoising with respect to the token positions. Specifically, they apply a lower movement speed to right-side tokens to guide models to reflect information in left-side tokens. Based on the dynamic movement speed method, they also propose a skipping mechanism during inference for efficient decoding. Experimental results show that the proposed method outperforms the previous diffusion-based approach at the same NFE, and the average performance drop is much lower in an extremely limited number of inference steps. In ablation experiments, they show that both AR-diffusion and the skipping mechanism are effective and the skipping mechanism can be applied to the other diffusion-based model.

Strengths

- The methodology is highly intuitive and well-motivated. - The proposed method is simple while mathematically supported and powerful. - They conduct experiments in various text-generation tasks to show not only consistent performance but also the efficiency of their method. - The skipping mechanism can be effectively applied to the other diffusion model.

Weaknesses

- Additional case studies comparing with GENIE or (N)AR models would provide further insights.

Questions

- Have you compared the decoding speed between AR-Diffusion with an inference step of 2 or 3 and NAR models? - Would AR-Diffusion also be robust to infilling tasks?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Since the model configuration of AR-Diffusion is based on Transformer-base in the paper, it would be possible to conduct a scalability study for various sizes in future work.

Summary

This work introduces left-to-right sequential characteristics into diffusion models, enhancing the text generation performance of diffusion models. By considering the AR model as a diffusion model with two states: to be decoded and already decoded, AR-Diffusion defines a continuous diffusion model with decreasing diffusion speeds from left to right. Experiments on various text generation tasks show that AR-Diffusion achieves improvements over existing diffusion models.

Strengths

1. The idea of introducing the left-to-right inductive bias into diffusion models for text generation is straightforward and reasonable. 2. By controlling the number of inference steps and generation candidates, AR-Diffusion can achieve tradeoff between quality and efficiency, which is more flexible than the Transformer. 3. Compared with the autoregressive model (BART), the generation of AR-Diffusion is more diverse

Weaknesses

1. I think the authors overclaim the decoding speedup. First of all, most diffusion baseline models in the paper have no advantage in both generation quality and efficiency compared with the Transformer. Thus, AR-Difffusion should compare with the Transformer for decoding efficiency. Besides, existing diffusion models can achieve competitive results with much fewer steps. For example, Difformer[1] and Dinoiser[2] can achieve competitive scores with 20 steps, and Diff-GLAT[3] can even generate high quality sequences with only 3 steps. Therefore, I think more comprehensive experiments should be conducted to claim decoding speedup. 2. Although the AR-Diffusion achieves better BLEU scores than that of the Transformer in Table 3, the results in Table 8 of Appendix C shows that the Transformer is still better than AR-Diffusion. Why are the results in the two tables contradictory? As SacreBLEU is a more standard metric for machine translation[4], does the results indicate that AR-Diffusion still lags behind the Transformer with a certain gap？

Questions

1. Which decoding method does the Transformer use, greedy search or beam search? If beam search is used, what is the beam search size of each reported Transformer result?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

AR-Diffusion requires a large number of candidates to achieve better results. Although generating hundreds of samples has large generation overhead, AR-Diffusion achieves promising results with MBR decoding. I think the large number of candidates is not a serious issue in the current stage. Reference [1] Gao, Z., Guo, J., Tan, X., Zhu, Y., Zhang, F., Bian, J., & Xu, L. (2022). Difformer: Empowering diffusion model on embedding space for text generation. arXiv preprint arXiv:2212.09412. [2] Ye, J., Zheng, Z., Bao, Y., Qian, L., & Wang, M. (2023). Dinoiser: Diffused conditional sequence learning by manipulating noises. arXiv preprint arXiv:2302.10025. [3] Qian, L., Wang, M., Liu, Y., & Zhou, H. (2022). Diff-glat: Diffusion glancing transformer for parallel sequence to sequence learning. arXiv preprint arXiv:2212.10240. [4] Post, M. (2018, October). A Call for Clarity in Reporting BLEU Scores. In Proceedings of the Third Conference on Machine Translation: Research Papers (pp. 186-191).